Post 2 of 3. Our IoT journey through ESP8266, Firebase and Plotly.js
TL;DR
My Medium home page is here.
This post stands in a series of 3 as detailed below. These 3 posts reflect the Project Architecture we chose to achieve a simple luminosity logging/live plotting project. We hope they will help developers or students discovering ESP8266 chip and Firebase platform. Moreover, we believe that the solution developped here can be an interesting alternative to the comprehensive Google, AWS and Azure IoT solutions [link, link and link] when we have to manage in a simple way only a few connected devices. (For instance, we won’t set up a MQTT broker neither deal with a device registry.)
Post 1 [link]: We investigate on how an ESP8266 can regularly make acquisition of an analog data and push its 10-bit equivalent value to a Firebase Realtime Database. We finish with considerations on ESP8266 power saving.
Post 2 (this article): We write a Firebase Cloud Function in Typescript language to timestamp the data just pushed to Firebase Realtime Database.
Post 3 [link]: With a web app using plotly.js library, we lively plot in a browser the analog data versus time. The web app is hosted with Firebase Hosting.
Post 2. A Firebase Cloud Function appends a timestamp to each value pushed to a Firebase Realtime Database.
A) The options we had for timestamping data
A) 1) In the previous episode…
At the end of Post 1 [link] of this terrific project, we regularly had a new 10-bit value of luminosity that was pushed into a Firebase RealTime Database — all done by an ESP8266:
As we want to plot luminosity vs time, a timestamp associated with each measured value is necessary. Indeed, we would like something like in the example below:
{
"timestamped_measures" : {
"-LHmx0njIsZ3hkCFzjSC" : {
"timestamp" : 1532010700350,
"value" : 910
},
"-LHmx1PcieyEm04aOo2a" : {
"timestamp" : 1532010702641,
"value" : 907
},
"-LHmx1leagyrTQxa9zii" : {
"timestamp" : 1532010703986,
"value" : 840
},
...
}Each measure should now be an object (described in JSON) having two key/val. pairs, the keys being "timestamp" and "value". And, at an upper level, this object is the value associated with the key called a “Firebase push ID”, for instance "-LHmx0njIsZ3hkCFzjSC" for the first record in the above example.
The timestamp of a measure should ideally corresponds to the Unix Epoch time of the instant the measure was made.
Note: The Epoch time is the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 [link]. Manual conversions from Epoch time to human time (or vice versa) can be made here [link] for instance. As you may have guessed it in the above example, we will use Epoch time expressed in milliseconds (not seconds) as the tools we’ll use supply timestamps within this unit.
We investigate now two options to associate a timestamp to a 10-bit value just obtained.
A) 2) First option: ESP8266 generates a timestamp and associates it to the just obtained value of luminosity
With this option, ESP8266 should be aware of time, i.e. have a Real Time Clock (RTC, hardware or software recreated). It should loop over the following actions:
- get the 10-bit luminosity measure.
- get Epoch time.
- create an object with
"value"and"timestamp"populated with the measure and the Epoch time. The ArduinoJson library we already installed in our previous article [link] can deal with that [link]. - push this object to our Firebase Realtime Database. This is possible with the FirebaseArduino library [link, link] we already know from our previous article [link]. The function for that purpose has this prototype:
String push(const String &path, const JsonVariant &value)Steps 1, 3 and 4 should be okay but step 2 is the tricky point. Indeed, when we read this NodeMCU Documentation at RTC Time Module page [link], we realize that RTC in ESP8266:
- is a software module,
- has an accuracy depending on chip temperature,
- needs to synchronize via SNTP or NTP (thus consuming power),
- will lose time in case of an unexpected reset.
Okay! Too much drawbacks! Let’s forget this option and let’s dive into a more realistic one…
Note: Using NTP with ESP8266 is nonetheless described here: [link].
A) 3) Second option: A Firebase Cloud Function issues the timestamp when a value is pushed to the Firebase Realtime Database.
Firstly, what are Firebase Cloud Functions?
Firebase Cloud Functions (or “Cloud Functions for Firebase”, [link]) are part of Google Cloud Platform and are functions executed in a Node environment on Google servers. A Firebase Cloud Function can be:
- called from an app involving Firebase (directly or via a HTTP request),
- automatically triggered upon an event on Firebase products:
You might have already guessed it, what we need is a Realtime Database trigger. Each time a measure is pushed to Firebase Realtime Database by ESP8266, a Firebase Cloud Function will be triggered in order to timestamp the measure. With that solution, we maintain the work done by ESP8266 to the minimum (and thus its power consumption).
The small drawback of this option is that we will timestamp the push to Firebase Realtime Database, not the measure. But the temporal gap between them is only about seconds, depending on network latency. In our previous article [link], we used to make a measure every 5 minutes ; a delay of 1 or 2 seconds on the timestamp is therefore not an issue here.
Note: Firebase Cloud Functions are part of what is sometimes called Functions as a Service (FaaS) and are one aspect of the famous serverless architecture, i.e. we don’t need to provide our own servers to host and run them.
B) Writing our Cloud Function
As written in https://firebase.google.com/docs/functions/ we should:
So let’s follow these steps! But before let’s give a few documentation links:
This page ↑ and its links often include videos from Doug Stevenson. We can find them on YouTube under the following list. It is worth watching all them, Cloud Functions are nicely explained and fundamental concepts like Promises too!
And finally, if you want to become a master in Firebase Cloud Functions 😃, head to this impressive Github repo of Cloud Functions samples (mostly in JavaScript):
B) 1) Setting up the tools managing Firebase Cloud Functions
To sum up :
Note: Each time there is an action to perform, it is preceeded by a lowercase letter like a. b. c. etc.
a. If not installed yet on our computer, we install Node.js [link]. It will also install Node Package Manager (NPM). Once done, we check version in command line. At the time of writing, it should be ≥8 for Node.js and ≥6 for NPM. To do so, open a terminal. Personnaly, as we develop on a Windows 10 computer, the terminal is cmd.exe. Do it right now with admin rights as some of the future commands will require it. Then, to check installed versions, run from any folder:
node --version
npm --version
b. We install Firebase Command Line Interface (CLI, a necessary set of Firebase command line tools) by running, from any folder:
npm install -g firebase-tools
Again, firebase --version will give the installed version. At the time of writing, it should be ≥4.
-g installs Firebase tools “globally” i.e. we will be able to use the Firebase tools from any project directory. A project directory is a directory on our computer that hosts during its edition the Cloud Function(s) related to the Firebase project we’re working on.
c. We authenticate with Firebase services by running (a few permissions will be asked):
firebase login
Note: Here, we have to authenticate with the same account than the one we used while setting up esp8266-rocks Firebase project in our previous article [link].
Note: Use first firebase logout if you were logged-in with another account.
It should result in this:
d. In our computer development directory (c:\_APP for us), we create a project directory that we call esp8266:
c:\_APP>md esp8266 on Windows (or mkdir esp8266 on Linux).
We navigate to it and run the command to set up the “functions” purpose of the project (same commands for Windows/Linux):
c:\_APP>cd esp8266
c:\_APP\esp8266>firebase init functionsThis is what we get:
e. A few questions are asked after we respond Yes to proceed:
- Of course, we choose to associate the
esp8266project directory with theesp8266-rocksFirebase project:
- We then choose TypeScript language. Usually programs for Node environment are written in JavaScript but TypeScript has a lot of advantages over JavaScript, the first one being likely (optional) static typing. In fact, it handles the pillars of Object Oriented Programming. However we have to confess that our function will be so simple today that using TypeScript will provide no real benefit... But the scaffolding for writing later more sophisticated functions in Typescript will have been set up. A short and nice introduction to TypeScript is here: [link].
Note: Check later this blog post [link] from Doug Stevenson for TypeScript pros while writing Cloud Functions. Read also [link] for additional workflow details while using TypeScript with Cloud Functions. You will also learn the key differences between a file in TypeScript and its equivalent in JavaScript ES6 (import <-> require, export const <-> exports.)
- We answer Yes to the next question concerning the use of TSLint. It is a powerful debug tool but unfortunately we won’t have time to tackle with it in this article! At least it will have been set up.
- At last, we accept to install dependencies with NPM: (be prepared to have a bunch of files installed!)
f. Now we’re done! On the graph below we have an (incomplete) tree of the project directory. What’s important to us is that inside the esp8266 project directory, a functions directory was created. In this one we find a src directory that hosts index.ts , the file where we will write our Cloud Function in TypeScript. At the time of deploying the Cloud Function on Google servers, the index.ts file will be transpiled to JavaScript, by a transpiler program called tsc, resulting in the index.js file stored in the lib directory. It’s index.js that will be run by Node.js on Google servers.
The node_modules directory hosts the dependencies (as Node packages/modules) indicated in package.json: firebase-admin and firebase-functions. But it also stores the dependencies of these dependencies… That’s why it is so huge.
esp8266/
|
+- .firebaserc
|
+- firebase.json
|
+- functions/
|
+- package.json
|
+- src/
| |
| +- index.ts # where we will write our Cloud Function
|
+- lib/
| |
| +- index.js # result of index.ts transpilation
|
+- node_modules/ # directory where dependencies (declared in
# package.json) are installedB) 2) Writing the Cloud Function
We launch an editor on our project directory. For instance, Visual Studio Code [link] is launched over the esp8266 folder that way:
c:\_app\esp8266>code .This is what we get:
There is already some (commented) code in index.ts. This is in fact the code of a function called on a HTTP request, not what we need today (but it’s worth testing it along with the first YouTube video embedded above in this post). Remember, we are about to write a Realtime Database trigger function.
Note: If we needed to write several Cloud Functions, they would all fit in index.ts, one after one.
We copy/paste the following code in index.ts. It is fully commented to understand how it proceeds. Of course, we save this file!
B) 3) Deploy, test and monitor the Cloud Function
Cloud Function deployment
In terminal, we run:
c:\_APP\esp8266>firebase deploy --only functionsUnfortunately, there is a known issue while predeploying (i.e. tslint & tsc execution) with Windows 10 (cmd.exe):
This issue is discussed here : [link, link]. We used this sustainable workaround:
- Replace
firebase.jsoncurrent content:
{
"functions": {
"predeploy": [
"npm --prefix \"$RESOURCE_DIR\" run lint",
"npm --prefix \"$RESOURCE_DIR\" run build"
]
}
}by:
{
"functions": {
"predeploy": "npm --prefix functions run lint && npm --prefix functions run build"
}
}- Then, replace in the
"scripts"part ofpackage.jsonthe lines:
"lint": "tslint --project tsconfig.json",
"build": "tsc",by:
"lint": "./node_modules/.bin/tslint -p tslint.json",
"build": "./node_modules/.bin/tsc",We try again firebase deploy --only functions and this time it’s ok! By curiosity, you can have a look at the obtained index.js file.
Note: It takes about 30 seconds for Cloud Functions to be deployed on Google servers. As we will repeat many times the write-deploy-test process, deploying Cloud Functions locally during development could be an interesting option. We can find more information on Cloud Functions emulator here: [link].
Cloud Function test
In the animation below, we trigger the Cloud Function. To achieve this, we simulate in Firebase Console a new measure pushed to Firebase Realtime Databasemeasures node. Instantly, the Cloud Function detects this creation and push to the timestamped_measures node the object with value and timestamp keys we talked about before. It also assign it a unique push ID. The value of the timestamp is the Epoch time in ms provided by Google servers, it should be accurate.
Cloud Function monitoring
While debugging, it is useful to monitor the execution of the Cloud Function and to console.log() a few informations as we did at the end of our index.ts file:
console.log(`Detected new measure ${original} with pushId ${pushId}`);Where can we see those logs? In Firebase Console (https://console.firebase.google.com):
How to undeploy a Cloud Function?
Curiously, it’s not possible to delete a Cloud Function from Firebase Console. We have to go to Google Cloud Platform Console (https://console.cloud.google.com), then to choose “Cloud Functions” and then to select the Cloud Function we want to delete:
How many times can we invoke our Cloud Functions per month?
With Firebase “Spark pricing plan” (the free one), we can make up to 125K invocations per month in the limits of 5K invocation per day and 50 invocation per 100 seconds [link]. This thresholds can be quickly reached! To know the current number of invocations (it’s easy to estimate it), we hit Project Overview in Firebase Console:
Note: When the invocation quota is exceeded, the function execution is paused and timestamps will be the ones when the function execution resumes! We should detect the “quota exceeded” event and /or consider the “Blaze pricing plan”.
Hmmh… wait! Do we have to change database access rules ?
First, remember that the Cloud Function we wrote has admin rights and can thus do anything with the database regardless of the rules we’ll set up.
But it’s good practice to set access rules for each node we have in the database, knowing also that .read and .write rules cascade to deeper paths (see [link]).
So, for our new node named timestamped_measures, we set .read and .write to false as, for the moment, no third party needs to access this node. But this will change in the next episode [link].
{
"rules": {
"measures": {
".read": "false",
".write": "false"
},
"timestamped_measures": {
".read": "false",
".write": "false"
}
}
}Conclusion
In this article, we wrote our first Firebase Cloud Function to timestamp measures made by ESP8266. No doubt you will adopt them in your own projects as it is an easy way to run code that cannot be run by client side.
As live data is now presented in a good shape, we will plot it dynamically with a web app using plotly.js in the third and last article of this series. Hopefully we’ll post it before Christmas 2018 🎄 🎁! So stay tuned!
Links to the posts: Post 1, Post 2, Post 3, GitHub and Live Demo.
Check also this post: GCP-Cloud IoT Core with ESP32 and Mongoose OS.
