InfluxDB — Migrating from v1 to v2

https://pixabay.com/photos/data-computer-internet-online-www-2899901/

Double trouble — migrating OS and influxdb at the same time

In most cases, simultaneously doing more than one complex and time-consuming task is not recommended since it is at least twice as hard to troubleshoot if things start to go south. But sometimes you can’t avoid it. In our case, the main reason was the fact that we were facing substantial downtime (in hours) for a service used by many, primarily as a data source for Grafana dashboards.

Goes without saying, but don’t just semi-automatically use the commands below. Look at them as a list of steps for people who prefer example commands instead of lengthy descriptions.

Before terminating CentOS 6, let’s take some notes and do some prepwork. Make note of databases, users and their privileges.

influx -username 'admin' -password 'xxxx' -execute 'SHOW DATABASES'
influx -username 'admin' -password 'xxxx' -execute 'SHOW USERS'
influx -username 'admin' -password 'xxxx' -execute 'SHOW GRANTS FOR a_user'

Also, make note of admin user passwords since they won’t be migrated automatically.

First we want to back up everything associated with the influxdb service. To start, create a backup directory.

mkdir /influxdb/backup && chown influxdb:influxdb !$

Next, back up the metastore.

sudo -u influxdb mkdir /influxdb/backup/metastore
sudo -u influxdb influxd backup /influxdb/backup/metastore

Get the list of databases you’ve obtained from one of the previous steps (e.g. db1, db2, db3) and back them up as well.

sudo -u influxdb mkdir /influxdb/backup/data
for i in _internal db1 db2 db3;do sudo -u influxdb influxd backup -db $i /influxdb/backup/data/;done

Back up configuration and WAL files.

mkdir /influxdb/backup/config
cp -rfp - parent /etc/influxdb /influxdb/backup/config/
mkdir /influxdb/backup/wal
cp -rfp - parent /var/lib/influxdb/wal /influxdb/backup/wal

Lastly, back up the data directory.

mkdir /influxdb/backup/data-dir
cp -rfp - parent /data/influxdb /influxdb/backup/data-dir/
cp -rfp - parent /data/log /influxdb/backup/data-dir/

Now you can proceed with the installation of Rocky OS 8 (or your OS of choice). In this example, we have a separate /influxdb partition used to back up all the data which will be used to store new influxdb2 data as well, so ensure that it’s not overwritten and that it is properly mounted after the installation.

The procedure on how to install Rocky OS itself is out of scope of this article, but you shouldn’t have any trouble finding one.

Once you have successfully installed a new OS, we can start having fun with influxdb. First, install the same version it was running on CentOS 6.

yum -y install influxdb-1.7.6–1 collectd

Restore configuration:

cp -rfp /influxdb/backup/config/etc/influxdb/influxdb.conf /etc/influxdb/

Restore metadata:

sudo -u influxdb influxd restore -metadir /data/influxdb /influxdb/backup/metastore

Restore databases:

for i in _internal db1 db2 db3;do sudo -u influxdb influxd restore -metadir /data/influxdb -datadir /influxdb/influxdb -db $i /influxdb/backup/data;done

Enable, start and check status:

systemctl enable influxdb.service
systemctl start influxdb.service
systemctl status influxdb.service

Clean up:

• Remove any unused (0 measurements/series) databases
• Remove any non-admin user without any grants
• Revoke any grants for users on non-existing dbs

Start the migration to the v2 process by stopping the service:

systemctl stop influxdb.service

Remove the service:

yum remove influxdb

Install influxdb2:

yum -y install influxdb2 influxdb2-cli

Prepare the configuration file /etc/influxdb/config.toml which could look something like:

bolt-path = "/influxdb/influxdb2/influxd.bolt"
engine-path = "/influxdb/influxdb2/engine"
http-bind-address = ":8086"
reporting-disabled = true
tls-cert = "/location/of/existing/server.crt"
tls-key = "/location/of/existing/server.key"
cp -pf /etc/influxdb/influxdb.conf.rpmsave /etc/influxdb/influxdb.conf
chown influxdb:influxdb /etc/influxdb/influxdb.conf

Run upgrade. The estimate is that it can upgrade around 1 TB in 30 min. Last pre-flight checks:

/influxdb/influxdb /data/influxdb /var/lib/influxdb/wal exist
/var/lib/influxdb/.influxdbv2 /influxdb/influxdb2/influxd.bolt don't
mkdir /influxdb/influxdb2
chown -R influxdb:influxdb /influxdb/influxdb2
sudo -u influxdb influxd upgrade -m /influxdb/influxdb2/influxd.bolt -e /influxdb/influxdb2/engine

> Welcome to InfluxDB 2.0!
? Please type your primary username admin_v2
? Please type your password ********************
? Please type your password again ********************
? Please type your primary organization name RL-Rulez
? Please type your primary bucket name some-bucket
? Please type your retention period in hours, or 0 for infinite 720
? Setup with these parameters?
  Username: rl_admin_v2
  Organization: RL-Rulez
  Bucket: some-bucket
  Retention Period: 0

Have a coffee or a brew since it’ll take a while. Once it’s completed, hopefully without too much nail biting or at least with enough left so you can type, run the following command and start the service:

systemctl start influxdb.service

It’s never a bad idea to have the CLI set up and ready. First, get the token:

grep -A3 default /var/lib/influxdb/.influxdbv2/configs | grep token | awk '{print $3}' | sed 's/"//g'

Then, create the configuration:

influx config create -n local -u https://your_server_name:8086 -t PASTE_THE_TOKEN_HERE -o RL-Rulez -a

NOTE: use set instead of create to update.

And finally, autocompletion:

influx completion bash > /etc/bash_completion.d/influx && source !$

In case the firewall is blocking port 8086 from your office network or VPN, you can try something like:

ssh -L 8086:localhost:8086 your_server_name
Access on https://localhost:8086

Sample configurations

Telegraf

[[outputs.influxdb_v2]]
  urls = ["https://your_server_name:8086"]
  token = "dontbesilly"
  organization = "RL-Rulez"
  bucket = "some-bucket/autogen"

Kapacitor

[[influxdb]]
  enabled = true
  default = true
  name = "influxdb"
  urls = ["https://your_server_name:8086"]
  username = "kapacitor"
  password = "againdontbesilly"
  timeout = 0
  insecure-skip-verify = false
  startup-timeout = "5m"
  disable-subscriptions = true
  subscription-mode = "cluster"
  subscription-protocol = "http"
  subscriptions-sync-interval = "1m0s"
  kapacitor-hostname = ""
  http-port = 0

Grafana DS — Custom HTTP Header named token has to be added. Everything else should remain the same.