Home Automation

Python, IoT: random thoughts

Introduction

For a while now, I have been playing with automating my home. I even wrote my own software, AutoBuddy. It works quite well, and this has given me some insights I feel like sharing with you.

Home Automation

At a very low level, home automation/IoT systems are just communication buses. The communication bus carries events (A switch is flicked, a presence is detected, a temperature changes,…). A home automation system simply gets those events and, if needed, responds to them.

By essence, all those events are asynchronous, they can happen at any time. When developing a Home Automation system in Python, the use of asyncio is therefore quite natural.

Although events are the most common messages, when developing AutoBuddy, I came up with 5 types of messages:

• Events, some entity reports a change of state, these can occur at any time
• Commands, instruct a connected entity to change its state e.g. switch turn on, light goes red, play music, … In most cases, once a command is carried out, an event will be generated.
• Requests, ask an entity to perform some task on your behalf, e.g. log some data, save a configuration, perform some calculation, ….
• Responses, returns the result/status of a request
• Restricted Events, an event containing sensitive information, e.g. password change.

This is by no mean the only way, but I have found these 5 types sufficient to carry out all the tasks needed by my home automation system.

The Bus

As mentioned previously, the type of system we are talking about is first and foremost a message bus. We need to choose a technology to implement the message exchange.

1. AMQP : AMQP would seem to be the perfect choice. It is supposed to be designed for the very type of application we are talking about. There are a number of implementations, Qpid and ActiveMQ from the Apache Foundation, RabbitMQ and others. In Python, py-amqp is a library to deal with AMQP. Unfortunately, AMQP suffers from a major drawback, versions 1.0 and pre-1.0 are not compatible, so, in effect AMQP is a standardized mess. Note that, initially, AutoBuddy was using AMQP, but I moved away from that.
2. MQTT : MQTT is also a seemingly perfect choice for this type of application. The protocol is nicely defined and offers all the functionalities needed. It can also be used out-of-the-box by multiple devices. Unfortunately, many of the devices that can use MQTT natively are low-powered devices that cannot use MQTT over SSL. This led me to use MQTT for devices that could use it, but not as the main medium for the message passing bus.
3. CoAP : CoAP is meant for machine-to-machine communication. It is good for low powered devices, but it is not well suited for a message broker.
4. Ad-Hoc : Message passing is not a fundamentally complex task, you have one server, a number of clients connect to the server, tell what messages they want and the server echoes the messages received according to what they want. Python with asyncio makes it easy to write TCP server/clients with SSL. This is what I ended up choosing. It has the added advantage that the message broker could be made to provide some services, like persistence (i.e. saving things to a database)

Connecting to the bus

Now that we have a message bus, we need message producing/consuming entities. In AutoBuddy these are called Buddies. More commonly they are called agents.

1.Bridge Buddies

Bridge buddies connect the message bus to actual devices. They communicate with the device with whatever protocol is necessary, MQTT, CoAP or device specific protocols.

Bridges usually require other libraries to communicate with the devices. In many cases, there were no asyncio libraries available so I had to write/adapt quite a few of them, for instance

1. aiolifx was adapted to communicate with Lifx LED light bulbs
2. aioblescan was written to get data from BLE sensors, initially Ruuvi tags.
3. aioxiaomi is for communicating with Xiaomi Yeelights

2.Proxied Buddies

Proxied devices are a special category of devices that exist only through a proxy device. For instance, an air conditioning is often proxied by a remote control device. In this case a bridge would control the remote control device ( Broadlink, Tasmota) and proxied device would be tied to a specific remote control. Devices like A/C, Curtains, AVR and all Zigbee devices would be handled this way.

3.Service Buddies

Service Buddies provide services. It could be provisioning service, Zeroconf service detection, … Since the point is to have a home automation system, an automation service is expected.

4.Device Buddies

Devices buddies are controlling a single device. VoiceBuddy is such an agent, it controls a microphone and enables voice recognition. In most cases, device buddies would not run on the main server.

Devices

There are many devices available. LED light bulbs, switches, buttons, locks, sensors, curtains, valves, and many more. The thing you need to be aware of is how to connect them and how to control them.

A cursory look at what is available will make quite clear that most companies provide their own cloud-based solution. Now, I am not against the use of the cloud, and in some cases that is the best solution, but I do not see the point of using an external server to control devices in my house. Another issue is that by using the manufacturer’s application, you send them your data. I don’t know you, but I’d rather not send my data to a company/government.

In order to control your devices locally, you need to be able to connect to them and know how to communicate with them. This, it turns out, is not always simple.

Connecting Devices

There are several technologies available to connect to your devices, each with pros and cons.

WiFi

This is the easiest. By now (2021) everybody has a WiFi network at home, so connecting a device to your WiFi network is quite easy.

Pro

• Easy
• No need for extra hardware

Cons

• With many devices, you need to have more hardware for WiFi coverage and may need to have more than the 254 IP addresses available in your standard configuration.
• Provisioning is usually not secure. It often relies on the devices setting up an access point, your telephone/computer joining that WiFi network and sending your WiFi SSID and password to the device. In my experience, all devices provisioned this way will be over a non encrypted WiFi network. In the case of Tuya devices, it is even less secure as the encode your WiFi password in the length of broadcast packets (reversed engineered by codethteweb).
• Many manufacturers do not publish the communication protocol and rely on their devices phoning home to be used with their app.
• Most devices still use IPv4.

Infrared/Radio Frequencies

Infrared and radio frequencies remote control are omnipresent. They can be used to control your devices

Pro

• Cheap
• Common

Cons

• Need an extra device to send IR/RF commands
• One way communication. You can not ask the device state and this can easily lead to discrepancies between the system view of the device state and its actual state.

Zigbee

Although I have not used the technology much, it seems really good when devices are properly implemented.

Pros

• Very cheap
• Long lasting battery-powered devices
• Mesh technology, powered devices can relay messages to the so-called coordinator over long distances.
• Open, the specification is available at the Zigbee website.
• zigpy is a Python implementation of the protocol.

Cons

• Need an extra device as coordinator
• Recommended to have not more than 70 devices by network. Can be alleviated by having more than one network.
• Interference with WiFi network. Alleviated by careful selection of WiFi and Zigbee channels. In most cases use Zigbee channels 15, 20 or 25.
• Some weird devices implementations. For instance, IKEA’s Trådfri Motion Sensor behave like a Zigbee switch even though Zigbee defines a motion sensor device.

Z-Wave

I have no experience with Z-Wave, suffice to say it is similar to Zigbee minus the interference problem.

Bluetooth/BLE

Bluetooth is a well supported, mature technology. It can use very little power

Pros

• Cheap
• Long lasting battery-powered devices
• Possible mesh technology. (proposed standard)
• Well supported in Python (PyBluez or native sockets)

Cons

• Many manufacturers do not publish their communication protocol and rely on a connection to a phone application to work.

Discovering Device

In many cases, devices can be automatically discovered. A home automation system should really be able to discover the devices connected to your network. If using a MQTT connection, the device will simply appear, but there are other methods. Some devices (Lifx, Yeelight,…) have specific methods, in AutoBuddy, the corresponding Buddies will handle the discovery. For others, they can be discovered via ZeroConf/Avahi/Bonjour or other technology. In AutoBuddy, ScannerBuddy offers that service and enables tracking devices IP address changes which can occur when using DHCP. It can be used to track Kodi instances, Denon/Marantz devices via their Heos service, etc…

Provisioning

One of the most tedious activities of setting up a home automation system is provisioning, that is configuring new devices to join your system.

As mentioned before, for WiFi, it consists mostly of connecting to a temporary access point provided by the device and configuring it using some web interface. It is so tedious that I wrote a python library/application to help me configure my devices, aioiotprov. In some cases, (Tuya, Xiaomi) the provisioning will return a key that will be used to encrypt all subsequent communication with the device.

For Zigbee, one has to set up the coordinator and devices into pairing mode. The devices have then to be held very close to the coordinator for the pairing to occur. By requiring the devices to be held close to each other, it improves the odds that the sensitive information needed for the pairing (PAN id and key) won’t be leaked.

For Bluetooth, the provisioning depends on the device. The Ruuvi tag sensors I use do not need to be provisioned, they simply broadcast the information over BLE in a publicly available format.

Alternative Firmwares

Many devices, especially chinese devices, are trying to force you to send your data to the manufacturer. In that case, the local protocol is not published and they will only publish a cloud interface. In the case of Yeelights, they do publish a local interface that can be enabled through the application, but newer bulbs seem to still require a Internet connection even for the local protocol to work.

One thing many of those WiFi connected devices share is that they use the ESP8266 or similar chipset. They can be flash with Tasmota, an alternative firmware that will free you from the manufacturers heavy tactics. Tasmota allows you to control your device over a Web interface and MQTT. The people developing Tasmota are trying to make it as easy as possible for us to flash the alternative firmware, even with OTA flashing. Manufacturers, on the other hand, keep trying to make it harder. Checkout the list of supported devices before you try flashing Tasmota.

Open Systems

There are quite a few open sources home automation systems available on the Internet. Since I wrote my own system, I did not use them, but they do have many contributors, so they are probably easier to set up and use than my software.

Home Assistant

Home Assistant is a Python home automation system. It supports many devices. It uses my aiolifx library for controlling Lifx LED lights.

Domoticz

Domoticz is written in C++.

openHAB

openHAB is written in Java

You can find more references here.

Recommendations?

I do use quite a few devices. Here are the devices I like to use. I have no connection to the respective companies except as a customer.

Light Bulbs

1. Lifx, good quality but pricey.
2. Shelly Bulb not quite as good as Lifx but good.
3. Zigbee Bulbs. I have not yet integrated them, but they do look OK.

Switches

1. Shelly, these are actually relays, you add them to your existing switches. Also sensor reporting electrical quantities.
2. Zigbee
3. Sonoff and similar flashed with Tasmota

Sensors

1. Zigbee
2. Ruuvi tags, for environmental

Remote Controls

1. Broadlink. Some like the Pro have both IR and RF
2. Tuya flashed with Tasmota

Curtains/Drapes

I mostly use Dooya motors with batteries that can be controlled by radio frequencies. It works very well. I also use a Shelly switch configured for that purpose.

Conclusions

Home Automation does not have to be expensive. The market today is a lot more interesting than when I started over 5 years ago. If I were to start today, I would go with

• A RaspberryPi 4 running Home Assistant
• A Sonoff Zigbee coordinator flashed with Tasmota. Do set the channel, PAN id and key before starting to pair devices
• A bunch of Zigbee and Shelly devices
• A few Broadlink IR/RF remote to control curtains and A/C’s
• Shellies for switches

Article written by François Wautier, organizer of PyCon APAC 2021

For more information on PyCon APAC 2021, visit our website