Thanks for publishing an interesting article!
OpenTSDB-protector is good solution against OpenTSDB limitations. Its’ ideas can be used for other TSDBs out there.
In general, OpenTSDB has too many limitations comparing to more modern TSDB systems such as M3DB, InfluxDB or VictoriaMetrics:
Did you try using purposely-built time series database additionally to or instead of PinalyticsDB? There are many good choices with various pros and cons out there — InfluxDB, TimescaleDB, M3DB, Cortex or VictoriaMetrics. There is also ClickHouse, which also can be used for time series workloads.
There are no such plans yet. If you feel certain PromQL extension should be ported to Prometheus, then just create Prometheus issue describing this extension, so Prometheus devs could decide whether it should be added to Prometheus.
While flushing every second increases the number of compactions to be done, the summary disk IO bandwidth usage is still smaller comparing to systems that use WAL, since the data stored in sstables has better compression ratio comparing to data from WAL. See disk usage graphs at https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b .
Thanks for the interesting article with many technical details! Recently GitLab posted an interesting article about anomaly detection. I like their approach due to simplicity and clarity. What do you think about it?
Also which time series databases do you use? I work on VictoriaMetrics TSDB, so I’m interested in this question :)
label_values function in Grafana — https://grafana.com/docs/features/datasources/prometheus/#query-variable . The function populates values for the given label, so they can be used as template variables.
Thanks for the article with interesting technical details!
It is interesting to know the following additional details:
I’d recommend trying VictoriaMetrics additionally to Wrap 10 as time series data storage. It has the following features: