Hello Zigbee World, Part 0 — Motivation, Goals, Plan

My story

Many years ago I was reading Bill Gates’ “The Road Ahead” book. The book described a concept of a smart house, where lighting, temperature, and even ambient music was controlled with a computer algorithm. The smart home could adjust to the tenants’ schedule, or even mood, providing maximum comfort and convenience. The only problem is that this system was very expensive in those years.

The situation changed around 2018, when I found a youtube video presenting Xiaomi Aqara smart home kit. I was an experienced engineer by that time, and possibly could create some Arduino based solutions for smart homes, but this required a lot of work. At the same time Xiaomi devices provided a ready to use solution out of the box. Unlike Arduino based devices that looked like a clew of wires, Aqara ones had a pretty nice design, and devices were factory produced. Moreover the starting kit did cost less than $100.

I bought a couple of Xiaomi Aqara switches and several sensors, and I began to enjoy life in a slightly smarter apartment. Later I switched to zigbee2mqtt plus a CC2538 stick, which allowed me to integrate devices from other manufacturers to my system. Overall, I was inspired with confidence in the future of the technology. When moving to a bigger apartment, I ordered a dozen more Xiaomi Aqara switches and looked forward to how cool it would work on a scale. I was naive….

The bummer came in the first week after the move. Red error lines started appearing in zigbee2mqtt logs. Switches strangely and randomly fell off the network, stopped responding to Z2M commands, as well as sending their status to the system. Switches themselves basically continued working as switches, but no longer were controlled by the system. Some switches could turn off spontaneously, and sometimes turn on when you try to turn them off. But the most annoying thing was that some of the temperature and occupancy sensors near the switch also disappeared from the network. Power juggling didn’t help. Once a day, they had to make an evening round and re-join the switches that had fallen off — only this could bring them back to life. After some experiments I was pretty sure that the problem was in switches, but other device types may have issues as well.

It appears that this problem is known, and I could just buy switches from another manufacturer which do not have this issue. Moreover Xiaomi presented new versions of their switches, that probably have this problem fixed as well. It would cost me a few more hundred dollars and forget about the problem.

But I decided to go with a longer, but much more interesting way. I wanted to get a better understanding of what is going on, and possibly fix the problem myself. I uncovered the sniffer, and started diving into the ZigBee. Xiaomi devices are based on the NXP JN5169 microcontroller, so I started looking at those datasheets as well. The result of the dive was the reverse engineering of the device, and some confidence that I could write an alternative firmware which would fix mentioned issues.

The Goals

As I just said the goal is to develop an alternative firmware for Xiaomi Aqara QBKG12LM switches. This is the first model of Aqara smart switches, has two buttons and a neutral line. Besides its connectivity issues, I lacked a few modern features. So I state the following list of features of an ideal QBKG12LM switch firmware.

• Stable operations in the large Zigbee network
• Possibility to bind switch channels to other devices (e.g. relay switches or lamps)
• More precise control on single/double/long clicks
• More precise control on decoupling internal relay switch from the buttons
• Integration with zigbee2mqtt
• Keeping other functions (temperature and power measurements) working

Obviously this requires a deep dive into Zigbee networks, and a good understanding of how this works under the hood. Having this experience I could also work on other devices’ firmware, such as temperature, light and occupancy sensors, power measurement devices. Zigbee knowledge could be also helpful when developing firmware for other Zigbee microcontrollers used in other manufacturers’ devices.

Unfortunately I have not found a simple Zigbee tutorial on the internet. There are a few projects like ‘I have an alternative firmware — use it’. There are also a few open source projects that provide some firmware skeleton with a short instruction on how to extend it. But these projects do not provide me enough confidence in the technology.

So here I come to an ultimate goal — create an extendable platform for creating Zigbee devices. This ambitious goal includes:

• Creating easy to understand example code
• Creating a set of articles and tutorials that explain Zigbee fundamentals, as well as the code
• Share articles with DIY community
• Ideally this would include not just NXP JN5169 microcontroller, but possibly other MCUs as well

The Plan

The QBKG12LM switch is a quite complex device. It has a lot of on board circuits that I do not fully understand at the moment. So there is a quite high risk to do something wrong, and break the device. Moreover, this is a line powered device with a high voltage line, so there is also a risk of injury, or damaging a connected computer.

I was also looking at a development board based on the same NXP JN5169 microcontroller. But the official development board from NXP costs $650, which is too expensive for experiments. Fortunately there is a E75–2G4M10S module from EBYTE that costs just a few dollars. I bought several modules for around $3 each. So it is quite cheap for experiments, and I would not get upset if I burn a few. The only problem is it is just a module, and requires some soldering to make a circuit. But this is not a big deal.

Next big problem is that the examples provided by NXP are too complicated for a newbie, and poorly described in the documentation. Usually when I learn a new technology I am working in small increments, attacking one problem at a time. Moreover the best approach to get a deep understanding of a technology is to start a simple project from scratch, and add features one by one. This allows me to control each step, and quickly get back to a previous step if something goes wrong.

Before I can get to the real Xiaomi hardware, I would like to practice on that module first. For this purpose I’ll try to build a simple device that would act like a smart switch. This will allow to develop all the building blocks that could be helpful for the real device firmware:

• Join the network and get properly identified by zigbee2mqtt
• A few buttons that will generate zigbee switch commands
• React on on/off commands and light a corresponding LED
• Handle single/double/long button presses
• Binding of the switch with other Zigbee devices
• OTA firmware update
• Custom zigbee clusters for controlling the device settings
• Temperature and power measurements, that are available in the Xiaomi device
• All these additional capabilities shall be integrated with zigbee2mqtt

When this is done I can easily port this to the real Xiaomi hardware. Moreover these building blocks will be also helpful for creating firmware for other Xiaomi devices.

After some experimenting, and having some progress on the project, I can now define a plan.

Stage 1 — get a working device using E75–2G4M10S module as a standalone device

• Solder a simple circuit with a few buttons and LEDs (Article)
• Set up the toolchain, that can compile the firmware (Article)
• Write a simple application, e.g. LED blinker (Article)
• Get understanding of how to flash the device (Article)
• Learn basic peripherals, that could be used in the project e.g. GPIO and UART (Article)
• Use watchdog timers (Article)
• Understand building blocks provided by SDK, e.g. software timers and message queues (Article)
• Understand power saving options and sleep modes (Article)

Stage 2 — get a working device as a Zigbee network node

• Get basic understanding of Zigbee Network and SDK (Article)
• Initialize Zigbee stack (Article)
• Join the network (Article)
• Register device in zigbee2mqtt (Article)
• Implement a basic Zigbee smart switch functionality (Article)
• Handle re-joining the network in case of network loss (Article)
• Implement direct binding to other devices (Article)
• Understand the difference between reports and commands (Article)
• Handle single/double/long button presses. Support in zigbee2mqtt (Article)
• Create a custom Zigbee cluster in order to expose device settings (Article)
• Make a custom zigbee2mqtt external converter to handle the custom cluster (Article)
• Implement OTA firmware update from the device side (Code ready, article pending)
• Add support for OTA firmware update for DIY devices in zigbee2mqtt (Code ready, article pending)

Stage 3 — nice to have stuff

• Using C++ in the project (Article)
• Extendable structure of the project codebase (Article)
• Understand how end devices work (Article) and how they handle joining and rejoining to the network (Article)
• Control dimming lights via Level Control cluster (Article)
• Explore how to build a dimming light (Article)
• Automated testing of the smart switch functionality (Article)
• Explore Identify cluster (Article)
• Explore Groups cluster (Article)
• Remote flashing of the device mounted on the wall, alternative to OTA update (Code ready, article pending)

Stage 4 — porting to the real Xiaomi QBKG12LM hardware

• Reverse engineer the schematics (Done, article pending)
• Implement joining/leaving the network
• Port switch function to the device
• Drive LEDs on the board
• Drive switch relays
• Handle temperature sensor
• Handle current and power sensor
• Evaluate possibility to switch from stock to the custom firmware via OTA update
• Test the device in house
• Share the firmware with Smart Home community for a wider testing

Stage 5 — evaluate other MCUs

• Evaluate NXP JN5189 (ARM based one)
• Evaluate possibility to use modern C++ compilers
• Evaluate possibility to use FreeRTOS
• Evaluate Texas Instruments chips
• Evaluate BL702 chip

The half of the project is already done. Usually some learning, coding and experiments go first, and when I have enough material I write an article. So the articles go slightly behind the code.

Support

This project is being developed for free as a pet project. At the same time you may consider supporting the project with a small donate.