Over-the-Air Arduino firmware updates using Firebase, Part 2
In which we support multiple firmware versions.
If you’re like me, you are incredibly lazy. This means not wanting to run around plugging cables into your Internet-connected Arduino devices when they should be able to pull new firmware over the air. For the cloud-controlled LED light display on my house, I have six Arduino boards on my front porch, and I’ll be damned if I’m going to go outside in the rain and plug my laptop into each of them whenever I want to push a firmware update.
This is Part 2 of a series of articles on how to use Google Firebase to support over-the-air firmware updates for an Internet-connected Arduino project. In Part 1 (which I highly recommend you read first), I provided a basic setup for storing Arduino binaries using Firebase Cloud Storage, and sample Arduino code to update the firmware over the Internet.
In this article, I’m going to talk about how to support multiple firmware versions and have your Arduino devices automatically determine when to update themselves.
To accomplish this, we’re going to leverage some other features of Firebase to maintain some metadata about the firmware versions we’re supporting.
What firmware version am I running?
In order for your devices to know whether they should update their firmware, they need to know what version of the firmware they are currently running, so they can compare it against the “desired” firmware version maintained in Firebase.
There are many ways of doing this. For example, you could define a string in your Arduino C code representing the version, like so:
#define FIRMWARE_VERSION "1.0.3.2.beta7.alpha4.zeta981892"If you go this route, you need to be careful to update this string in your code every time you have a new version you want to push out to your devices (before compiling it). Personally, I’m way too lazy to remember to do this, and wanted an automated solution.
It turns out that the compiler can automatically fill in a “version identifier” for you when it compiles your code. The magic compiler strings __DATE__ and __TIME__ represent the (human-readable) date and time at which the binary was compiled, and can work perfectly well as a version string that is automatically updated every time you compile. So, you can do something like this:
const char FIRMWARE_VERSION[] = ("__MaGiC__ " __DATE__ " " __TIME__ "__");When you compile the sketch, FIRMWARE_VERSION will contain a string like "__MaGiC__ Feb 05 2019 16:15:21 __" which is great as a unique identifier for this exact build of your firmware. (Just be careful about things like timezone changes, leap-seconds, going back in time, etc. to ensure you don’t end up with two different builds with identical version strings!)
The magic string __MaGiC__ will be used to automatically extract the version string from the binary later on — see below.
Storing firmware metadata in Firebase Realtime Database
The next thing we want to do is keep track of all of the firmware versions we have uploaded. For each firmware binary, we want to keep track of several pieces of information: Its version string (from above), the date and time it was uploaded to Firebase (in case it differs from the build time), and the URL where it can be retrieved from Firebase Cloud Storage. You might also want to keep track of other things, like a short description of the firmware (such as “This is the version that works” or “This is the version that REALLY works”).
For this, we’re going to store a record in Firebase’s Realtime Database for each firmware binary. Check out my earlier article about using Firebase to control Arduino devices over the web for a primer on using Firebase’s Realtime Database. What we want to do is store a record for each firmware binary with the following fields:
dateUploaded: The date that the binary was uploaded to Firebase.filename: The human-readable filename (e.g., “AwesomeProject.ino.bin”)url: The URL where the binary can be retrievewd from Firebase’s Cloud Storage.
The key for each database entry is going to be the version string for that binary (e.g., "Feb 05 2019 16:15:21").
Now, you could of course manually enter all of this information in Firebase Console, under the Database tab. But what a pain!! Instead, what we’re going to do is write a little web app that lets you upload a binary image and store all of the corresponding metadata in one fell swoop.
Uploading firmware using a Web app
As described in my earlier article, the Firebase Javascript API makes it easy to write web apps that read and write data to Firebase. In this case, we need a web app that does three things:
- Allows the user to upload a local file.
- Extracts the version string and other metadata from the file.
- Uploads the file to Firebase Cloud Storage.
- Stores the corresponding metadata to Firebase’s Realtime Database.
Okay, that’s four things, but you get the idea.
Step 0: Firebase config for the web app
You’re going to follow the same basic setup for a simple Firebase web app as I have documented before — include the following boilerplate (which is specific to your Firebase project!!) in your main HTML file:
<script src="https://www.gstatic.com/firebasejs/5.8.2/firebase.js"></script>
<script>
// Initialize Firebase
// NOTE!!! The below is specific to your Firebase project --
// use "Web Setup" from the Firebase "Develop" pane to get this
// code for your app.var config = {
apiKey: "AIzaSyABCaBBY04zpvIl1efmOPrKwNtPkgTXfqs",
authDomain: "team-sidney.firebaseapp.com",
databaseURL: "https://team-sidney.firebaseio.com",
projectId: "team-sidney",
storageBucket: "team-sidney.appspot.com",
messagingSenderId: "395332355872"
};
firebase.initializeApp(config);
</script>
Step 1: Extracting metadata from an Arduino binary image
Rather than ask the user to manually keep track of version identifiers, what we want to do is have our web app automatically pull the FIRMWARE_VERSION string out of the Arduino binary. This is where the magic string __MaGiC__ shown above comes into play. All we have to do is have our web app scan the uploaded binary file for this string and pull out the version. (This works as long as this string doesn’t appear anywhere else in your binary — if __MaGiC__ is not unique enough, I suggest using something like __BaNaNaSlUgS__ or __FiSh SlApPiNg DaNcE__ as alternatives.)
Depending on the web framework you’re using, you can use a simple <input> element with type=file to present the user with a file chooser UI:
<input id="uploadFirmwareFile" type="file" style="display: none;">You can then attach a change event handler to this object using something like this:
// Assuming you use jQuery:
$('#uploadFirmwareFile').change(function() {
var file = $('#uploadFirmwareFile')[0].files[0];
readFile(file);
});And finally, the JavaScript code for reading the file metadata and storing it to Firebase:
// Read the file chosen by the user.
function readFile(file) {
// Open the file and start reading it.
var reader = new FileReader();
reader.onloadend = function() {
readMetadata(file, reader.result);
}
reader.readAsArrayBuffer(file);
}// Extract the version string from the given file data.
function getFirmwareVersion(data) {
var enc = new TextDecoder("utf-8");
var s = enc.decode(data);
var re = new RegExp("__MaGiC__ [^_]+ ___");
var result = re.exec(s);
if (result == null) {
return null;
}
return result[0];
}// Called when we're done reading the file data.
function readMetadata(file, data) {
version = getFirmwareVersion(data);
if (version == null) {
console.log("Could not extract magic string from binary.");
return;
} // Upload firmware binary to Firebase Cloud Storage.
// We use the version string as the filename, since
// it's assumed to be unique. var uploadRef = storageRef.child(version);
uploadRef.put(file).then(function(snapshot) {
// Upload completed. Get the URL.
uploadRef.getDownloadURL().then(function(url) {
saveMetadata(file.name, version, url);
});
});
}// Save the metadata to Realtime Database.
function saveMetadata(filename, version, url) {
var dbRef = firebase.database().ref('firmware/' + version);
var metadata = {
// This bit of magic causes Firebase to write the
// server timestamp when the data is written to the
// database record.
dateUploaded: firebase.database.ServerValue.TIMESTAMP,
filename: filename,
url: url,
};
dbRef.set(metadata).then(function() {
console.log("Success!");
})
.catch(function(error) {
console.log("Error: " + error.message);
});
}
Whew! I know that’s a lot of code for a blog post (“I came here to read, not review GitHub pull requests”), but I hope it’s pretty self-explanatory. (Leave a comment if not.)
When this is done, our Realtime Database should have entries that look like the following:
iot-demo-3a7b9
|
+- firmware
|
+ __MaGiC 30 Jan 2019 09:38:56 __
| |
| +- filename: "AwesomeProject.ino.bin"
| +- dateUploaded: "30 Jan 2019 10:15:02"
| +- url: "https://firebasestorage.googleapis.com/....."
|
| __MaGiC 06 Feb 2019 16:11:23
|
+- filename: "AwesomeProject.ino.bin"
+- dateUploaded: "06 Feb 2019 16:28:40"
+- url: "https://firebasestorage.googleapis.com/....."Step 2: Telling devices which firmware version to run
The next step is to maintain some metadata in Firebase to assign each device a firmware version that it should be running. Devices can periodically poll this metadata to find out if the version they have is different than the configured version, meaning they need to do an update.
For this, we’re going to build upon the use of Firebase’s Realtime Database described in this article to store configuration data for each of our Arduino devices.
In particular, we’re going to store data such as the following in our database:
iot-demo-3a7b9
|
+- config
+
|
+- 30:AE:A4:1B:58:A0
| |
| +- version: "__MaGiC 06 Feb 2019 16:11:23 __"
|
+- 30:AE:A4:1C:1A:B0
|
+- version: "__MaGiC 30 Jan 2019 09:38:56 __"The keys for the config subtree of the database are the MAC addresses of each Arduino device that we are controlling — I chose these as they are pretty good as unique device identifiers, but you can use something else for this. The version entry in the subtree indicates which firmware version ID should be running on that device.
As before, you can use the Firebase Console to plug in these values by hand, but you can also do it using Javascript code from a web app, using something like the following:
var mac = '30:AE:A4:1B:58:A0';
var dbRef = firebase.database().ref('config/' + mac);
dbRef.set({
version: '__MaGiC 06 Feb 2019 16:11:23 __'
}).then(function() {
console.log('Success!');
})
.catch(function(error) {
console.log('Error: ' + error.message);
});Building a nice UI around this is left as an exercise for the reader.
Step 3: Updating the firmware on the Arduino side
Finally, to put everything together, our Ardunio devices need to periodically poll the config entry in the Realtime Database to find out which firmware version they should be running, and if it’s different, pull down an update.
#include <ArduinoJson.h>// Allocate a 1024-byte buffer for the JSON document.
StaticJsonDocument<1024> jsonDoc;void readConfig() {
String url = "https://iot-demo-3a7b9.firebaseio.com/config/" +
WiFi.macAddress() + ".json";
http.setTimeout(1000);
http.begin(url);
int status = http.GET();
if (status <= 0) {
Serial.printf("HTTP error: %s\n",
http.errorToString(status).c_str());
return;
}
String payload = http.getString();
DeserializationError err = deserializeJson(jsonDoc, payload);
JsonObject jobj = jsonDoc.as<JsonObject>(); const char *nextVersion = (const char *)cc["version"]; if (strcmp(nextVersion, FIRMWARE_VERSION) &&
strcmp(nextVersion, "none") &&
strcmp(nextVersion, "") &&
strcmp(nextVersion, "current")) {
startFirmwareUpdate(nextVersion);
}
}
Here we’re using the Firebase REST API to read the appropriate config entry from the database and parsing it as a JSON object to determine what the next firmware version should be. If it differs from the current version, we call startFirmwareUpdate() with the desired version string.
Next, we pull down the metadata on this firmware version from the firmware subtree of the database:
void startFirmwareUpdate(String firmwareVersion) {
String url = "https://iot-demo-3a7b9.firebaseio.com/firmware/" +
firmwareVersion + ".json";
http.setTimeout(1000);
http.begin(url);
int status = http.GET();
if (status <= 0) {
Serial.printf("HTTP error: %s\n",
http.errorToString(status).c_str());
return;
} String payload = http.getString();
DeserializationError err = deserializeJson(jsonDoc, payload);
JsonObject jobj = jsonDoc.as<JsonObject>();
firmwareUrl = (const String &)fwdoc["url"];
http.end(); updateFirmware(firmwareUrl);
}
Finally, we pass the URL we get back to a slightly modified version of the updateFirmware() function that I provided in Part 1 of this series:
void updateFirmware(String firmwareUrl) {
http.begin(firmwareUrl);
// Copy the rest from my earlier article...
}Summary
I’ve been using this technique to upload and push new firmware updates to my IoT projects for a while, and it works very well. Indeed, as we speak, I am laying in the sand in a tie and dress shirt (but no shoes or socks!!) happily pushing firmware updates to my Arduino devices. You should be envious.
Let me know in the comments if you have any questions or suggestions for improvement!
