Avoiding combinatorial explosion in continuous queries in Influx
How not to do CQs, and how to do them right
My small development environment suddenly stops producing performance data. I looked into few places and quickly realized that it’s influx and continuous queries it it are lagging behind. Influx consumed 130% CPU and had had a backlog for about of 5 minutes (with ‘every 1 minute’ CQ, means I have at least three missed CQs.)
My initial suspicion was to a newly connected application: ‘oh no they sends too much’. The issue persisted even after I killed that application, and dropped its measurements.
So I had to dig deeper.
I omit my attempts and guesses. Here the answer: NEVER use regular expressions in CQ for measurement names.
Long explanation
The official Influx documentation provides the following example of CQ:
Example 3: Automatically downsampling a database with backreferencing
Use a function with a wildcard (
*
) andINTO
query’s backreferencing syntax to automatically downsample data from all measurements and numerical fields in a database.
CREATE CONTINUOUS QUERY "cq_basic_br" ON "transportation"
BEGIN
SELECT mean(*) INTO "downsampled_transportation"."autogen".:MEASUREMENT FROM /.*/ GROUP BY time(30m),*
END
cq_basic_br
calculates the 30-minute average ofpassengers
andcomplaints
from every measurement in thetransportation
database (in this case, there’s only thebus_data
measurement). It stores the results in thedownsampled_transportation
database.
cq_basic_br
executes at 30 minutes intervals, the same interval as theGROUP BY time()
interval. Every 30 minutes,cq_basic_br
runs a single query that covers the time range betweennow()
andnow()
minus theGROUP BY time()
interval, that is, the time range betweennow()
and 30 minutes prior tonow()
.
This is bullshit. Do not trust them.
If you have more than one measurement in a database, and those measurements have different fields/tags, you will have a combinatorial explosion during CQ execution.
You can see this by yourself by executing this query:
SELECT mean(*) FROM /.*/ GROUP BY time(30m),*
If you have those (many measurements), you will see a crazy output with lines and lines of field names, a very long list of tags (all tags combined!) and screens and screens of empty space with occasional values.
The reason for that it that Influx combines for SELECT
part all fields (from all measurements) together, and do grouping by all tags (from all measurements).
Somehow INTO
part of CQ deals with this and places everything as it should be. Nevertheless, there is a combinatorial explosion (Cartesian product?) for all possible values. Even on very fast machines it takes eterninty to crunch this madness. And it getting exponentially worse with each next measurement you add (a few more items into telegraf plugin list). That’s why I hadn’t notice this eariler. I added just four new plugins, and my test installation dropped from respectable ‘everything is OK with CPU < 10%’ down to miserable ‘lagging behind and hogging CPU’. Just four simple plugins! I glad it had happened in my lab, and not in a production.
What to do?
Answer is a bit sad, but it allow to tame CQs. Instead of few CQs handling all measurements in database, create a per-measurement CQ. With few retention policies it cause another combinatorial explosion, but it’s much more contained (as it’s a product of usually fixed number of retention policies and o(n) from number of measurements).
The main downside of all that is that you loose ability to automatically process new measurements with a single CQ. Each new measurement need few more CQs.