Weather Driven Boiler Automation Done “RIGHT” (boring details)

This is the boring part of the main post:

https://medium.com/fundbox-engineering/weather-driven-boiler-automation-done-right-d62ba2dbbd4e

Design

Some clippings of the internal dialogue:

• Why Google Calendar? — After some thought, it’s the best pragmatic “UI” solution. Any custom UI I would make would probably be worse, and I use the calendar all the time anyway.
• Why cron? — Because I don’t want to deal with a running process, it doesn’t feel RIGHT — smells of memory issues, PIDs, soft restarts, and all that jazz.
• Timer API vs. Google Assistant. — Hopefully, the timer’s API exists and works; I would prefer it as it needs fewer hops and has less Google.
• This “AI” sounds dumb. — Yes, it is. All it should do is figure out how many hours are needed given the weather. The main reason to dumb it down — a more intelligent AI might endanger future me’s peace of mind. But if it won’t be enough, I will improve it later.

A note from future me: the design was good but far from perfect — it’s not detailed enough, e.g. the process of getting the weather shouldn’t be coupled to the main logic! Making it later a stand alone data fetching job made much more sense.

Sniffing

It took ten minutes to verify that there was no API available for my timer. It took twenty more minutes to quit the denial and search for a cli solution to trigger Google Assistant. There was none. That dark, sinking feeling of realization that I’ll have to research a complicated corporate library. I won’t go without a fight, though!

A note from future me: this was BAD sniffing! There is no official API for this timer, but someone did write a python client that works great on LAN. I ended up using it, although it would have saved me only a day of work: I wanted to receive broadcast notifications when the boiler goes on/off anyway.

Google assistant POC adventure

I quickly found the official code sample that knows how to send a text command to Google Assistant. It has a dozen of arguments, interactive user input, and a grpc loop. When I see code with even one of those, I immediately try to avoid it by using the skill of procrastination, during which I look for other, less nausea-inducing solutions.

I won’t describe the two unpleasant hours I spent trying to find an alternative. There was none. I lost the fight.

Google ecosystem, here I come.

A note from future me: yep, I found later such alternative, there is a docker image that sets up an http server that sends text messages to the assistant. But! It’s a good thing I missed it — implementing this code myself instead of dealing with dockers on a Raspberry Pi is much better.

There is this decent tutorial. In this tutorial, you’ll experience: some links that direct you to some wrong places; some mentioned page elements that are missing; some steps in some other kind of English than you know. Same as most tutorials. Manageable.

Most of it — bureaucracy (I had to do every step in the list below multiple times):

• Create a project in Google Cloud Console
• Create OAuth credentials
• Set up a consent screen
• Register the device in Actions Console
• Download the client credentials file
• Run a google-oauthlib-tool to get the actual credentials

Three failed projects, two leaked credentials, and one project that might probably be ok — I could run the sample script! I could start digging inside! (I must acknowledge it was a happy moment, although low value for money)

I had to do a lot of cutting and slicing, removing the extra bits, converting arguments into a configuration file, etc. But it was not that bad. I did require a (cold) shower, though.

Interesting findings:

• Making it trigger some action was easy; getting a response if the action was successful was not.
• The response has multiple parts and depends on the device capabilities; e.g., the answer will lack a “text response” if the screen_mode setting isn’t set.
• Some responses are never transferred as text, only as audio.
• The answer to the question “is the boiler on?” comes only as audio.

But! For what was supposed to be a “yes, it’s on.” The audio stream content is different than a “no”! I didn’t want to deal with voice recognition — so, I just md5ed it and hoped google doesn’t change its responses. It’s not RIGHT, but I can’t do anything feasible about it.

This is unfortunate, future me — I’m so sorry.

A note from future me: I was so naive, it blew up in his face back then, never reached the future me — the “voice recognition” part was so unstable that I had to search again for some other solution, this is when I found a proper client for my timer. I do still use google assistant to broadcast the boiler state though. I wish I could get rid of this too, I already had two incidents of revoked tokens and I have no idea why.

Research

How do I calculate how much time is needed to heat a boiler?

Let’s break it down into smaller problems:

1. How much kWh is required to heat 1L of water?
2. How much kW is my boiler? What is the capacity?
3. What is the current temperature of the boiler water?
4. What temperature do I want the water to be?

How much kWh is required to heat 1L of water?

First, I started thinking about areas, volumes, and heat exchange. But then I remembered about the internet and found the answer right away (linear equation makes sense as heat is energy, my mistake was to think about it in terms of flow):

kWh = 4.2 × liters × temp. delta ÷ 3600

How much kW is my boiler? What is the capacity?

Thanks to my currently installed timer — my boiler consumes 10A.

I think decent people who don’t work with electricity shouldn’t know Ohms law by heart — internet to the rescue:

kW = Ampers × 220 Volts / 1000

I won’t even calculate it; I’ll just stick the formula in the code for clarity.

Boiler amperes: 10A

Capacity (a short trip to the roof gave me the answer):100L

What is the current temperature of the boiler water?

A note: this was written before I found visualcrossing

This is the least known out of the unknowns. My first approach was to get a Bluetooth-enabled water thermometer that I would put somewhere along the pipeline. There were two problems with that: the only place in the pipeline that I could intervene was the showerhead, and by then, the water was already mixed. And on top of that, I couldn’t find such a thermometer anyway. (My initial idea was to create a neural network and feed it the data from some wireless-enabled measuring device, but I decided not to pursue that direction as I didn’t want to become that measuring device.)

So I ditched the source of truth and went for the factors that affect it. The calculations will be less precise, but we are not boiling <insert dish name that needs precise boiling temperature> here. Some temperature differences are alright (I took a massive leap of faith here as the other shower user is my wife, a very picky showerer).

A note from future me: she hasn’t noticed anything. Yet.

What affects the water temperature in a solar boiler? Well… The sun. In less capacity, the air around the boiler. Which is affected by the sun after some delay anyway.

A note from future me: this is a lie. Reality is much more complicated of course.

But how much kWh is the sun? Surprisingly wiki had a comprehensive answer:

In the tropics insolation can be relatively high, e.g. 7 kWh/m2 per day, versus e.g., 3.2 kWh/m2 per day in temperate areas.

I started to think about how to take into account the date of the year to calculate the sun’s luminosity but then realized that it doesn’t matter for my needs; I can just deduce it from the air temperature.

A note from future me: this is also a lie.

The contraption:

• When the temperature is under a specific value, we assume zero sun power
• When the temperature is above a specific value, we assume maximum sun power
• When it’s cloudy, we multiply the sun power by cloudiness percentage
• We hope that all of these correlate nicely and linearly and hope that it’s close enough to reality so it’ll have some predictive value.

Yes, I know the math here is not based on reality, but the numbers make sense, and I can continually improve it — tweak the formula or add new data sources.

A note from future me: I continually improved it by deletion and replacement with a better data source.

The formula:

What temperature do I want the water to be?

A quick search gave me that a “hot shower is somewhere between 37–41°C. This means it needs to be somewhat hotter so the water can be mixed. Also, I need to be able to heat to a lower temperature to have some creative control over the schedule. I chose a range of 30–55°C.

Classes Design

The repo: https://github.com/devenv/better_boiler_automation

Scheduler

The class.

This is the glue that holds everything together. All the other classes are dependency injected so that it can be tested easily. The logic is simple:

• Get the schedule and the weather data from the store.
• Ask the Calculator how many hours are needed to heat the water.
• Tell the BoilerController to switch on or off accordingly.

Of course, all the logic is thoroughly tested, including some simulations:

CalendarSync

The script.

Another Google sample I just tweaked to work. Retrieves events from a calendar and saves them in a format Scheduler understands.

WeatherProvider

The class.

The only thing you need to know about getting weather information with python is that pyowm is awesome, has excellent documentation, and was a ray of light after the abyss of Google Assistant SDK Experience. You need to get your free API key from OpenWeather first. Getting the weather:

The weather data I used: the temperature and the cloudiness for the last N hours.

A note from future me: beware, OpenWeather is wrong when it comes to some data. e.g. clouds — OpenWeather, I can see the sky, why do you tell me it’s 100% clouds? I ended up using visualcrossing which has sun energy metric and their report about clouds correlates with my observations through the window.

BoilerController

The class.

It is a really thin abstraction layer between the Scheduler and the assistant.

It also provides a test-mode logging-only controller; instead of calling the assistant, it will log the actions and maintain the state.

Calculator

The class.

Calculates how many hours need to heat the boiler given the weather.

Let’s talk about testing.

I could skip writing tests till now:

• Assistant — my brain blacked out right after I started figuring out how to write tests for the Assistant class. It’s not my code, and if I ever have to change it, I’ll have to test it manually vs. the real deal anyway.
• CalendarSync — same (except I did write a test to make sure the contract with the Scheduler won’t break)
• WeatherProvider — can be tested with some mocking, but it’s too simple and probably will never change. I promised myself if I’d ever touch it in any meaningful way — I’d write tests.
• BoilerController — it’s just a layer, no actual logic.

The only reason I allowed myself to call this RIGHT is that I work on this project alone. Otherwise, these excuses wouldn’t cut.

Anyway, in the Calculator case, there is some actual logic. Writing this without tests would not be RIGHT, and would not be easy.

Store

FileDataPersister

The class.

Knows how to store strings.

StackDataPersister

The class.

Knows how to store multiple strings in a stack (limits the number of values stored).

FreshDataStore

The class.

Knows how to store a stack of dataclass values and the timestamps when the values were added. It can then retrieve only the fresh ones (e.g. for the last six hours)

ScheduleDataStore

The class.

Implementation of FileDataPersister stores the current schedule.

TemperatureDataStore and CloudsDataStore

The class.

Implementation of FreshDataStore[float], stores the last 100 weather values.

Assistant

The class.

The monstrosity I created earlier.

Switcher

The class.

Just a wrapper around aioswitcher.

Raspberry Pi

After some birth pains, my code worked on a Raspberry Pi on python 3.7 (I had to downgrade my code from 3.10). Raspberry Pi is fantastic, but it’s slow.

Installing google assistant packages was a big mess. e.g., I couldn’t pip install because one of the requirements was the grpc library which requires more than 300Mb of RAM to build, not something I could afford at the moment. Eventually, I found that library’s binaries for armv6 architecture and compiled the rest.

Datadog Agent Raspberry Pi Issues

This is the longest step in the whole project. It took a week and had no actual value. Not even educational. Sad.

The ways Datadog agent didn’t work for me:

The official installation instructions

• Ended up in a core dump.

The other official installation instructions

• A note at the bottom: Datadog does not officially support Raspbian.
• Succeeded, but the agent version was too old.
• No logs, no traces.
• I ended up using this version until I moved to Grafana.

The unofficial docker

• Worked by taking hostage most of the resources.
• I couldn’t make my project work in a docker as compiling the Google assistant library in a docker is even less feasible.

Other solutions from the depth of the internet

• Compiling newer versions of the agent: at some point, the compiling was nine hours in and I had a feeling it was nowhere near the end.
• Compiling in a virtual machine on a proper PC: felt like an intentional attempt to make the rabbit hole deeper.

Each step took multiple days as Raspberry Pi is slow and doesn’t have much memory.

Grafana Agent

I installed all that was needed by following the official instructions. It took a day to set everything up and add Prometheus support.