Measuring, Decision Making, and Outcomes

Making good decisions is all about understanding the context in which they are being made, and measuring their outcome. I’ve been thinking a lot about this lately, in the context of having a smarter home, and around improving mental health. We need to be getting the right data to the right systems with the correct understanding of that data in order to make inferences from it, implement changes, and verify we’re having a positive impact.

Figuring out the correct things to measure, and accounting for them accurately, is a huge part of the job. It’s no good making changes based on bad data. It’s also not ideal to be making arbitrary changes when you can’t measure the success of their outcome — to know what conclusions we can draw from the results of the experiment.

I’d like to run you through a simple example that I’ve been thinking about for a few years now, but which I haven’t yet managed to run any well structured experiments on (mostly because it involves hardware hacking, which is not my forte!). I have a Rancilio Silvia coffee machine which I use to fuel myself with coffee in the morning. It can pull a pretty good espresso, but the results can be a bit variable at times, and I’d like to get better consistency.

My hypothesis is this: pulling the perfect espresso is all about the water hitting the group head at just the right temperature. (The group head is just above where the puck of ground coffee sits, tamped, patiently waiting to have hot water pushed through it.) There are, of course, many other variables — like the water pressure, the volume of coffee in the basket, and the coarseness of the grind, never mind the choice of beans themselves, their freshness, etc — but a good experiment keeps those elements constant and varies just one aspect.

The trouble with this little experiment is around measuring the outcome. It’s all down to the taste of the finished product after all, which is a terribly subjective measure, and not one that’s even consistent over time. But we’ll just have to cope with that, and hope that I can at least be consistently subjective when I’m tasting the results of each experiment.

(That’s something I’ve been thinking about a bit, too. For lots of experiments — even ones with objective measures — it’s not the absolute value of the measurement that’s important, it’s about the change, or even rate of change over time. And we often get hung up on measuring the wrong things, then optimising for these measurements, rather than the desired outcome. I see this a lot around software development process “improvement” for example… But I’ll get into these topics in more depth in another post!)

Right now, the temperature of the water hitting the group head in my coffee machine is pretty variable. The machine has a small (300ml I think) copper boiler. There’s a electric heating element in there to apply heat, and a thermistor attached to the boiler. When the temperature of the boiler drops below the lower threshold, the element is switched on, which heats the water in the boiler. When it reaches the upper threshold, the element is switched off again.

The nature of the current measuring system is such that the outcome — the temperature of the water hitting the group head — is wildly variable. (In fact, I don’t know this for sure, so my first experiment should be to measure that!) Under different circumstances — the machine being switched on first thing in the morning; when it’s had some time to warm up; or how it changes in reaction to different environmental temperatures — I’d expect slightly different results, but from a bit of Googling in previous years, I’m anticipating the variance being of the order of 30ºC which, for ‘boiling’ water sounds spectacularly bad. My understanding is that the ideal temperature for brewing coffee is about 92ºC, so that’s what we should be aiming for.

One thing we can do is to improve (reduce) the range of temperature for the water in the boiler. We can do that by improving the measuring device so we’re getting a more accurate picture of the current temperature, and by improving the control system for the element. I’m a bit hand wavy on the measuring device — I’ve no idea if it’s currently a thermistor or a thermocouple, or something even more primitive! — but I’ve an idea that we can improve it, so that we can infer an accurate temperature to make control decisions from.

That’s an interesting aside, in fact. Oftentimes what we can measure, and what we want to measure, are subtly different things. In the case of a thermistor, we’re wanting to measure the temperature of something. In reality, what we’re able to measure is the electrical resistance across the thermistor. However, thanks to our understanding of the laws of physics, we’re able to infer a good enough approximation of what we want to measure from what we’re able to measure. This applies to all sorts of things we want to collect data on, though we often don’t have the knowledge to make such strong inferences!

Our current control system is pretty simplistic. If the temperature drops below our lower tolerance, we apply heat. If it exceeds our upper tolerance, we remove the heat. The trouble with heating a body of water is that there’s some inertia in the system. When the temperature drops below the lower threshold, it will continue to drop even after the heat is reapplied, before increasing again. And when the upper threshold is reached, and the heat is removed, it’ll continue to gain temperature (from latent heat in the element) for a short while before starting its descent again. The rate at which the temperature increases is governed by the amount of heat we apply, and the rate at which it decays will change depending on the temperature of the surrounding environment.

We can improve on the control system by using what’s called a PID (proportional, integral, derivative) controller. This controls the output (in our case the amount of heat applied to the system through the heating element) by figuring out the error value, which is the difference between current temperature and the desired temperature. It takes into account three things for the error value:

• The current temperature — the ‘proportional’ bit. So if the error is a large, positive value (the temperature is way over the desired value), then the output of the PID will be a large negative value (which in our system would just mean switching the heating element off entirely, but could in principle involve switching on some active cooling system instead).
• The historical temperatures — the ‘integral’ bit which, if I recall correctly from maths class, is the area under the graph you’d plot of the temperature over time.
• The projected future temperature — the derivative bit, which is the current rate of change. The faster it’s changing, the quicker it’s going to hit the threshold, and the further it’s going to go beyond the threshold before stopping and changing direction.

By taking all three of these into account, you can apply the right amount of heat to the system to keep it pretty consistently close to the desired temperature.

Of course, we’re making a pretty big assumption here: that accurately controlling the temperature of the water in the boiler will make sure that the water is the right temperature when it hits the coffee in the portafilter. I reckon that we can improve on that by measuring the actual temperature of the water when it hits the group head, and by measuring the variables that affect it — the current temperature of the group head, and the ambient temperature of the room seem like sensible measures. Given that information, and enough historical data, I reckon we can intelligently set the desired temperature of the water in the boiler such that it’s just right when it hits the coffee.

Once we’ve achieved the perfectly brewed coffee, we can branch into other experiments. I wonder, for example, how this control system affects the energy consumption of the coffee machine. Did our changes to improve the accuracy of the temperature pejoratively impact the energy efficiency? (My gut feeling — another hypothesis that needs testing! — is that it’ll be insignificant, but that it would have a slightly positive impact.)

All this is to say that I think measuring stuff is really interesting, and that in doing so, we can perform experiments that improve our lives. The trick is in finding the right things to measure, and in methodically performing these experiments, so that we can make good conclusions from the results. I reckon this systematic approach can be applied to pretty much anything, from automating your home, through improving business processes at work, and right along to improving your physical & mental health.

Then again, am I measuring and experimenting on the wrong thing entirely? Is the end goal to have an enjoyable coffee, or should my measure of success really be the impact on my productivity of that shot of caffeine? (Short answer: no! I’m doing this for the nice tasting coffee, and for the fun of tinkering!)

If you enjoyed this article, I’d really appreciate it if you recommended it on Medium (hit the ❤️ below) and shared it with your friends on Twitter. You might also be interested in reading about how I’m HomeKitting out the House and my HomeKit Goals.