Ice wine, Redis and Microcontrollers
This is part 29 of my Node / Redis series. The previous part was Grab your hardhat!
I started to get interested in the interface of software and the physical world in October of 2016. I had heard quite a bit about the Arduino ecosystem but lots of people were talking up Raspberry Pi and clones too. October 2016 was important in the Internet of Things world — loads of attention was drawn to the subject for a very bad reason: Mirai.
Mirai was (still is?) a botnet that infected internet connected devices — things like IP-connected security cameras and DVRs. On October 21, 2016 it caused a major outage at Dyn and people started to take notice. This was an attack by a thousand cuts, while the amount of processing power in a DVR or camera is tiny compared to a laptop or web server, a coordinated DDOS attack made from thousands of IoT devices was fairly trivial because the security was so lax. The reason? No one ever wants to update an IoT device and, sometimes, it’s not even possible.
Back to my consideration of software and the physical world — the idea of building devices based on Linux started to give me heartburn. Linux is fantastic but you can’t say it’s simple. Frankly, I don’t understand every nook and cranny of the OS, so securing it is non-trivial — I will not, even remotely, account for everything. In a situation like a server, I know I can go back and make changes but if you build a device and sell it to 100,000 people, there is no going back and you can’t count on customers/clients doing the right thing from a security update standpoint.
With that, I shifted my thinking and was reminded of the wise words of Colin Chapman: Simplify, then add lightness. Raspberry Pi definitely has it’s place, but for a great many uses, it’s overkill to have that much hardware and software. So, then, you need to look to the microcontroller world — where you design small, single-use software with no real operating system. In a practical sense, you’re throwing out as much as possible and using the least viable hardware.
The Arduino ecosystem is a miracle to my mind. It’s taken a very technical subject that was the bailiwick of hardcore engineers and made it accessible to even children. Arduino’s most popular board is the Uno. The Uno is powered by an 8-bit ATmega328 20MHz CPU with 2kb RAM. Let that sink in for a second. Think about those specs compared to the device from which you are reading this article. But yet, amazing things can be done with just this little bit of power. The reason being is that you are often doing one, simple thing with this tiny CPU — a small calculation or manipulating an input sensor to drive an output line. But the Arduino ecosystem is more than just the hardware. It’s the software as well — Arduino is based on a rich collection of libraries and an IDE that makes programming the devices fairly easy. Usually, all you need to do is plug in the device via USB and click the “Upload” icon. Finally, the Arduino community is vast and there is great documentation and a wide range of tutorials available.
Now, that I’ve gushed about Arduino and the Uno, let’s talk about a big problem. Internet connectivity is not built-in. You can achieve this type of functionality with an Uno by adding a shield but it’s expensive and, from what I understand, somewhat feature poor. Enter the ESP8266.
Gather round for the story of the ESP8266
Many people were having this same problem with the Arduino Uno — they had a neat device they wanted to add some simple internet connectivity to but the options for doing this were limited. A small module was floating around the various electronic supply sources — it was cheap (about 5 dollars at this point in time), small and could connect to WiFi. The interface was a frustratingly weird 8-pin package — the pin configuration meant that you couldn’t plug it into your normal prototyping breadboard.
The 8-pins of the package allowed you, with the prepackaged firmware to communicate over a serial interface with modem-style AT commands (anyone who spent time with a modem might remember ATDT to dial out). These AT commands could be used to scan for networks, connect and send data. All the usual stuff. It was made by a company called Espressif, who released very little information about it. This is where the story starts to get good.
On that module was more than just a WiFi modem, but rather a full-blown microcontroller. The microcontroller was a bit of an oddball — built on the Tensilica Xtensa architecture and clocked at 80 MHz. So, it was powerful. It also had 64 KiB of instruction RAM. Compare that to the ATmega in the Uno and you can see that using the ESP8266 as a WiFi co-processor for the Uno was a bit akin to letting a tow truck run a track day loaded with a Corvette. It doesn’t make any sense. The problem was that Espressif had very little documentation and most of it wasn’t in English. So, the eco-system was sparse due to lack of accessible documentation. But once the community realized the power of the chip and started translating everything from Chinese, the community blossomed.
What’s fascinating about the ESP8266 is that it’s a dirt cheap microcontroller (it’s now down to about $2 US per unit — cheaper than even a generic Uno clone) that can do quite a bit as it can connect to the internet. On top of this, there is a growing community surrounding it. My past few months dipping my toe into the microcontroller world has taught me that communities are everything when it comes to programmers — even the best product will die on the vine if there isn’t a community supporting it.
The other great thing about the ESP8266 is that you can actually program it with the Arduino IDE and many of the libraries have been ported over. It’s not exactly the same and the Arduino still is a more user-friendly microcontroller but it’s hard to ignore the ESP8266.
Enter Redis
With the 8266, my first thought was “Can I run Redis on this thing?” Obviously, we can’t run the redis server, as the amount of RAM would make it a rather useless thing, but what about as a client? If you have a Redis client then you can leverage Redis as a data store house for sensors or as a centralized data source triggering various real world activities to happen based on what’s stored at a key. You could even use a Redis module as a way of doing processing of data for the microcontroller. It’s a compelling thought, really.
You write software for an Arduino in a dialect of C/C++, honestly it’s just a preprocessor and a special few libraries but otherwise it’s gcc cross compiling for the different boards. So, knowing that hiredis is written in C and well used and test, I thought I’d give that a go. The go didn’t go very far. The ESP8266 network interface seems very different than what hiredis uses and it would be non-trivial to port from what I could see (I’m the first to admit that I’m not an expert at C. I can suffer through it at this point, but I quickly get over my head).
This brings me to one of the terrible things about the ESP8266 — it’s quirky. Internally, it uses a real-time OS (really, more like a BIOS, from what I can tell) in ROM to maintain the WiFi connection so you need to constantly either delay or yield, giving the OS time to do it’s thing or you’ll get a nice fat crash. A lot of the higher-level functions are written in ROM and it dictates how you need to write your code, but it means that many things, like WiFi are baked directly in. This explains why the IP stack is different than what hiredis uses.
Additionally, while the ESP8266 has a ton more RAM than the Arduino Uno, it’s still charitable to call the RAM size tiny. I was concerned that that hiredis would use too much overhead in this type of environment.
The Redis server communicates with the outside world using a protocol called RESP (REdis Serialization Protocol). RESP is pretty neat and human readable — it has a control character (+, *, — , $, and :) followed by either the data or the size of the data in decimal form(in the case of bulk strings which are followed on the next line) and a \r\n to end it. That’s pretty much it. You send commands this way and you get them back in the same format. So, I set forth to create a RESP parser and serializer in a language I barely knew and on new (to me) hardware platform.
The minimal viable RESP parser/serializer needs to be able to send commands and get data back out of Redis. I’m not looking for something that can maintain multiple connections, do pub/sub or even provide anything nice in the syntax. As naïve, crude and to the point as I can get away with in the smallest footprint I can muster. And, to note, I’m not calling this a client library — it’s mostly just a RESP parser and serializer. You might need to be a little more hands-on compared to more full-featured clients. But again, the goal here is to be able to read and write small amounts of information with Redis — that’s all that is required at this point.
Ice wine.
I’m currently in Ontario, Canada. It’s very much winter, but I think many outsiders would be shocked to know that Ontario produces 17.5 million liters of wine despite having a climate that is vastly different than other wine producing areas. And even in the winter, the wine harvest continues but not for your run-of-the-mill pinot grigio, but for ice wine. Ice wine is a sweet yet acidic libation made from grapes frozen on the vine. Recently, in a segment of the CBC radio program Ontario Morning, an ice wine producer (Georgian Hills Vineyards) was talking about the harvest. The segment starts at 27:45 and goes to 34:00
What caught my attention was the very specific temperature conditions which the wine has to be harvested — the temperature must be between -8° to -10° C and it needs to be harvested all in a single morning. Small variations in the conditions and the harvest will fail — it’s risky. The payoff is high though — the tiny bottles of ice wine go for a premium.
When I think of processes that operate in small margins of success, my prescription is usually data. In the radio story, they use the weather forecast but as most people know, forecasts are pretty general and average out microclimates. What if the west side of your vineyard is -7°C and the right side is -11°C? The quality of your wine suffers. Could you, with extra data only pick the grapes on the trees that have reached the perfect temperature? Let’s build a prototype data collector based on the ESP8266 and some software interfacing Redis that an ice wine vintner could use to determine the perfect, per-tree conditions.
The hardware
The hardware will consist of an ESP8266 board, a temperature sensor, a resistor and a battery. Let’s just assume that the vineyard had WiFi (repeaters, LTE hotspots, whatever). The operating temperature of the ESP8266 is -40C ~ 125C, so we’re well within the operating conditions of the microprocessor.
The ESP8266 operates at 3.3v so, we’ll need to get some power to it. Most people might think about Lithium Ion or Lithium Polymer batteries due to their native voltages, but the charging of these cells can be complicated and my understanding is that these batteries might not perform very well in the temperatures required (although I’m no battery scientist). Indeed, I would not overthink it and employ a simple USB powerbank that is powered by AA Alkaline cells. Something like this powerbank from banggood (or eBay for slightly less $$).
As far as the module, I would go for something that is easy to program and relatively full-featured. For my everyday development ESP8266 board, I used the Adafruit Huzzah Feather — it’s expensive for an ESP8266 at $15.95 but it has a very reliable USB-Serial interface, on-board battery charging and a very bread-board friendly footprint. Also, Adafruit is a really good company that gives back to the community in many ways. You should support them where you can even if their gear is a little more expensive.
Deploying into an environment were you’re likely to program once and forget about it, I’d go for a cheaper board — the LoLin V3 NodeMcu board (again, I’m referring to Banggood because the link will be more likely to stay stable, but, at time of writing, the same board can be had on eBay for about $3.11, shipped). This board has the same ESP8266 module, but has a less-good USB-Serial interface, no on-board battery charging circuit and is too wide to actually use on a normal breadboard.
In our prototype I’m using a digital temperature sensor (about 2 bucks on ebay or banggood). I first went for a thermistor, but I found that it wasn’t super accurate or consistent in readings. After a little bit of research, I went for a sensor based on the DS18B20. It’s digital and uses the OneWire protocol to communicate with microcontrollers. Despite the name, the sensor has three wires, ground, power and data (there is a way to operate with so called “phantom power” but it’s more complicated). You’ll also need a 4.7k resistor (couple of cents from any number of sources) for this sensor.
First, I put it together a test on a breadboard (prototype of a prototype?). While this phase is useful to test out a concept, the resulting circuit is fragile and looks somewhat like a movie prop bomb. The observant among you might notice that I’m not using a NodeMCU board here but rather another ESP8266 variant, the WeMos D1 (which is also pretty good and has a stackable interface). In the upper right you can see the stainless steel end of the sensor and in the lower middle you can see the sensor with a soldered header to ease hookup.
The Development Software
For this project you’ll need to get the Arduino IDE and install the ESP8266 Arduino SDK. If you’re using the Adafruit Huzzah Feather, you’ll need to get the USB-Serial driver for the SiLabs CP2104 USB/Serial chip and for the LoLin NodeMCU board you’ll need the driver for ch340G (this can be tricky on a Mac). After you install the drivers, go ahead and installed the ESP8266 Board Package using the guide supplied by Adafruit. It’s pretty much the same for the Huzzah Feather or the LoLin, except for the selection of the port. Have I mentioned how great Adafruit is? They’re pretty great (and they don’t pay me to say that).
Once you’ve got all that installed, you might want to try to run the famous Arduino blink example code just to make sure you’re up and running.
As a software guy, hardware does seem crude sometimes. Some of the nice things we have when dealing with high level languages on powerful platforms just don’t exist in the hardware world, from what I can see. Case in point is getting your libraries integrated. This is the point where I would kill for something like NPM.
To install libraries in the IDE you’ll need go to Sketch → Include Library → Add .ZIP Library. Of course, that means you’ll need to download libraries in ZIP format. In this sketch, I’m using the following libraries:
- Arduino-Temperature-Control-Library (aka DallasTemperature)
- OneWire
- ESP8266 (sntp, ESP8266WiFi, WiFiClient, WiFiUdp)
You’ll also need to include your credentials in a credentials.h file. See the source code for specific examples
The Circuit
Below is the schematic of the circuit I’ve built. It’s pretty straight forward if you’ve ever played with a breadboard before.
I like using strip board (aka veroboard) for this type of simple circuit as it’s fairly close to how you treat a breadboard, but you can also look to making your own PCB. With stripboard, you have a conductor that runs the entire length of the circuit and you can remove portions of the conductor with a tool like a Dremel to make the connections you need. You solder the components on the conductor side. Here is my process:
The Code
You’ve probably been waiting for this part. The code we’re going to use is a fairly short Arduino sketch. The code needs to accomplish a few things:
- Connect to wifi
- Get the time
- Authenticate into your Redis server
- Get the temperature from the sensor
- ZADD the time and the temperature to a ZSET
- Go to sleep and start all over again.
Connecting to the wifi is not too tough. In this sketch we’ll just hard code the access point SSID and password. The ESP8266 has built-in EEPROM storage and people have developed very polished interfaces to allow a users to setup their network credentials, but it’s beyond the scope of this article. Andreas Spiess on YouTube has created some wonderful tutorials on how to setup the ESP8266 with a self-configuration web server (among other great ESP8266 related content).
Getting the time is a surprising challenge. You have a couple of options in this regard. The ESP8266 lacks a real-time clock, so you can either install a physical real-time clock module or utilize our internet connectivity. Since I’m a software guy, I’d rather not any more physical hardware than required, so I went with software. We’ll connect to an NTP server and use that to get a unix timestamp (code based on this script). This proved to be more complicated than I had imagined as NTP is UDP-based and gives out the time not based on the Unix epoch, but rather from Jan 1, 1900. That being said, it’s also not 100% accurate as the network transmission/latency is not accounted for so it might be a millisecond or two off, but good enough for our purposes.
Acquiring the temperature from the sensor is a matter of requesting the temperature from the library. As previously mentioned the sensor uses the fairly complex OneWire protocol and then needs further decoding. Thankfully, we can use existing libraries — OneWire and DallasTemperatureSensor (The DS18B20 is currently manufactured by Maxim Integrated Products that bought Dallas Semiconductor, thus the name of the library).
Now to Redis. My funky little RESP serializer/parser breaks out each step into individual function calls that need to be executed individually and operate on a struct for each call. First, you initialize the struct for your Redis command, then you add arguments. Internally, this is adding the RESP control characters and character counts to a string. Finally, you execute the command and it will collect the response back in the same struct. Being barebones, there is very little in terms of connection logic, so AUTH is just like any other command — pass in your argument and wait for a response.
To record the temperature we’ll be using a ZSET, but in a slightly unusual way. ZRANGEBYLEX has a useful feature — if you add multiple unique values to the same score you can use ZRANGEBYLEX get a range of results in lexical order. It’s a bit of a hidden gem of Redis, but it can be a little difficult to understand. Here is how our data is laid out:
ZADD temp-track 0 unix-timestamp:temperatureSince the unix timestamp rolled over to 1,000,000,000 in 2001 and will stay in lexical order until it reaches 9,999,999,999 in 2286, I think we’re ok for this application. Effectively, since we’re not going to record two temperature readings in the same second, lexical order will effectively ignore anything after the colon.
Finally, the microcontroller needs to go to sleep. There are a few ways to do this. In my prototype, I’m just using delay to stop activity for a minute. The ESP8266 also has a deep sleep mode. To use this you need to tie the D0 pin to the VU pin. This effectively shuts off the microcontroller for a fixed number of seconds. When those seconds have elapsed, then the ESP will turn itself back on (and start the entire script over again — it does not, however, resume at the same place with the same state).
Here is the Arduino sketch:
(At some point in the future, I might separate out the RESP parser bits to make them a legit library)
The web interface
This is the Node Redis series, so let’s build a small web interface to view the results of the temperature logging. To start, we’ll need to create our connection, using the familiar pattern of passing in the node_redis connection object in the arguments of the node script. We’ll use this to create the connection. Express will be our server framework — we’ll have a single route — this route will return the temperatures for a given rage of UNIX timestamps. The format of data will be a nested array that looks something like this:
[
[javascript timestamp, temperature in c],
...
]Note that in the response, we are returning javascript timestamps, which are the same as UNIX timestamps but in millisecond precision rather than second precision. To get our data from Redis into this format, we’ll map over the response and split the string by the colon. The UNIX timestamp is converted to a number and multiplied by 1000 to get the javascript timestamp.
Everything else will be served as static files. We’re using Angular 1.x, Chart.js library, ngTable, and Angular Chart plugin — all of these can be delivered via CDN. This means we have two files left: the HTML and our client-side server javascript. We can put these in the static directory and employ the static middleware built into Express to serve those out.
In our client-side JS, the controller for the single page sets up the options for the chart. The chart didn’t render the hover popup very nicely, so I swapped it out a more informative popup. Then we use the $http service to get the temperatures from the server. In this example, we’re always getting the last 24 hours of results, but it could be customized for any number or an arbitrary range from user input. Chart.js requires the data to be in a specific format, so we need to massage the output from the server into two arrays the data and the labels. I wanted to show the time along the bottom as the number of minutes since the current time, so I had to do the math. Finally, I setup the parameters for ng-tables to display and sort the data correctly. You can get the source at this github repo.
You should be able to fire up your browser to http://localhost:4999/ and see something like this:
Mircocontrol all the things
Microcontrollers are really at a fascinating point. You can have a tiny, inexpensive device that plays well with real, powerful tools like Redis. The simplicity and elegance of the RESP protocol complements network connected microcontrollers enabling them to leverage the power and capacity of Redis.
When you start thinking about the possibility of not only logging, but also sourcing data in the cloud (at, for example, RedisLabs, where you can get a free 30mb instance), microcontrollers bridging the gap to the cloud is very exciting. So, go out there, build something and hook it to Redis already!
