Two variables at a time: Taguchi arrays for speeding up experiments

Today I present a very unusual video, coming from a source I never imagined I’ll link to. It’s a hobby DIY ‘engineering for fun’ youtube channel which I never expected to give me big insights. But I got one.

As operator I often get to the ‘tuning problem’, when I need to change few important variables for a complex system (Ceph cluster, k8s, load balancer, etc) and I have no idea which is important (and in which direction). Should a window be larger? Smaller? Is larger MTU meaningful? If I reduce number of queues and length of the queue for NIC, will this change throughput or tail latency? Would many NUMA nodes works better than a fewer NUMA nodes for AMD servers?

Usually such experiments are limited to few variables, and have one change at a time, which is extremely time consuming.

This video discuss a technique to speedup such experiments, so called Taguchi arrays (orthogonal arrays). This is wiki link, if you want text representation: https://en.wikipedia.org/wiki/Taguchi_methods.

The wiki page is beyond my comprehension. Video is not:

(https://youtu.be/5oULEuOoRd0)

Tl; Dr;

There is a technique to have smaller set of experiments to find influence of variables on the outcome, which involves careful pairing of two variables at a time. With a proper pairing it’s possible to show which of two variables has an effect (or both, or one having negative and second having positive effect).

Why to watch?

Video gives two very concrete examples, and instead of reasoning why this is true, goes into properties of that method. It also discuss at layman level ‘why this is works’, but the main wow is that after the first artificial and boring example with egg peeling, there is an impressive real-life example where author of video is surprised by himself. That gives a strong emotional enforcement of the method, which would be otherwise a bit boring.

It also provide a ‘ready-made’ instructions about how to use it in practice, with that impressive second example, working as … example.