Doing SSL Requests on ESP8266 Correctly

For a home automation project, I recently decided to get a handful of D1 Mini with the ESP8266 WiFi microcontroller. Thanks to Arduino support and the number and quality of available libraries, setting everything up and deploying a small sketch was a breeze. There are tons of good step-by-step tutorials (such as this one) that explain how to get Arduino set up to work with a D1 Mini, so there’s no need to repeat that here.

However one thing drove me crazy: sending HTTP requests to an https web server. All the available examples and code I could find would specify the certificate fingerprint in the request in order to validate the server cert. This works okay for doing requests to a local server with a self-signed certificate, but I wanted my server-side code to run as GCP Cloud Functions, where short-lived certificates are used. So essentially, the fingerprint changes every 90 days, and my device would just stop working whenever the certificate is renewed.

Looking for a better way, I stumbled upon an example on how to use the BearSSL certificate store, but that one didn’t use ESP8266HTTPClient and instead assembled the request by hand which is kind of tedious.

In the end, I managed to put together a solution that combines the certificate store from the above example with ESP8266HTTPClient. Since you have the entire Mozilla root certificate store, it means your device can make SSL requests like you would in the browser, with certificates being automatically checked against the list of CAs that Mozilla trusts.

The first step of this solution is to fetch the certificate store from Mozilla and upload it to your ESP’s SPIFFS. The example above has a handy Python script that downloads the certificates and creates a “certs.ar” file for you. Copy this file into a folder called “data” within your sketch, then upload the file to SPIFFS. There’s good documentation available describing how to configure Arduino with a tool to do the upload for you.

Now that the certificate store is on SPIFFS, we have to load it into BearSSL. For this purpose, the BearSSL_CertStore Arduino example provides a helpful extension for the BearSSL CertStoreFile class that reads the store from SPIFFS.

#include <FS.h>
class SPIFFSCertStoreFile : public BearSSL::CertStoreFile {
  public:
    SPIFFSCertStoreFile(const char *name) {
      _name = name;
    };
    virtual ~SPIFFSCertStoreFile() override {};
    // The main API
    virtual bool open(bool write = false) override {
      _file = SPIFFS.open(_name, write ? "w" : "r");
      return _file;
    }
    virtual bool seek(size_t absolute_pos) override {
      return _file.seek(absolute_pos, SeekSet);
    }
    virtual ssize_t read(void *dest, size_t bytes) override {
      return _file.readBytes((char*)dest, bytes);
    }
    virtual ssize_t write(void *dest, size_t bytes) override {
      return _file.write((uint8_t*)dest, bytes);
    }
    virtual void close() override {
      _file.close();
    }
private:
    File _file;
    const char *_name;
};
SPIFFSCertStoreFile certs_idx("/certs.idx");
SPIFFSCertStoreFile certs_ar("/certs.ar");

Additionally, we also need to sync the time for the ESP with NTP, as that’s required for SSL certificate validation.

// Set time via NTP, as required for x.509 validation
void setClock() {
  configTime(3 * 3600, 0, "pool.ntp.org", "time.nist.gov");
  Serial.print("Waiting for NTP time sync: ");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
}

Now we can actually create our own WiFiClientSecure, where we can set the certificate store. A full example that sends a request via HTTPClient looks like this:

void setup() {
  Serial.begin(115200);
  WiFiManager wifiManager;
  if (wifiManager.autoConnect(ssid, password)) {
    setClock();
    SPIFFS.begin();
    HTTPClient http;
    BearSSL::WiFiClientSecure *client = new BearSSL::WiFiClientSecure();
    BearSSL::CertStore certStore;
    int numCerts = certStore.initCertStore(&certs_idx, &certs_ar);
    client->setCertStore(&certStore);
    Serial.println(numCerts);
    http.begin(dynamic_cast<WiFiClient&>(*client), "https://www.google.com");
    int httpCode = http.GET();
    Serial.println(httpCode);
  } else {
    Serial.println("Failed to connect to Wifi.");
  }
}

Finally, the ability to send arbitrary SSL requests and ensure the server has a valid certificate. I hope this helps people avoid having to go through the same rabbit hole I went.