Upgrading MySQL back-end for Gitea

These days we are working on upgrade procedures of our infrastructure components on Kubernetes. One of them is Gitea, as a back-end Git-registry for Argo CD. For Gitea, we choose MySQL as a database, where we started with 8.4.0, using the MySQL Operator and the MySQL InnoDB cluster Helm charts.

The problem

So, now we wanted to upgrade our MySQL Helm chart dependencies for MySQL operator and InnoDB Cluster to 2.2.2:

# Argo CD application Chart for MySQL Cluster
apiVersion: v2
name: mysql-cluster
description: An Argo CD Application Helm chart for MySQL Cluster
type: application
version: 9.1.0-2.2.2
appVersion: 9.1.0
dependencies:
- ...
- name: mysql-operator
  version: 2.2.2
  repository: oci://eu-amsterdam-1.ocir.io/axufzftra6ys/helmcharts
  alias: mysql-operator
- name: mysql-innodbcluster
  version: 2.2.2
  repository: oci://eu-amsterdam-1.ocir.io/axufzftra6ys/helmcharts
  alias: mysql-innodbcluster
- ...

Then upon refreshing Argo CD on the application and having the MySQL Cluster servers Cycling into the new version, the MySQL Cluster servers failed to initialize. I found the following in the logging:

[Entrypoint] MySQL Docker Image 9.1.0-1.2.19-server
2025-01-10T13:56:57.450181Z 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
2025-01-10T13:56:57.448554Z 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
2025-01-10T13:56:57.451459Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2025-01-10T13:56:57.451583Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2025-01-10T13:56:57.451694Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2025-01-10T13:56:57.451718Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2025-01-10T13:56:57.452101Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_group_replication_recovery_use_ssl=1'.
2025-01-10T13:56:57.452121Z 0 [ERROR] [MY-000067] [Server] unknown variable 'mysql_native_password=ON'.
2025-01-10T13:56:57.452129Z 0 [ERROR] [MY-010119] [Server] Aborting
2025-01-10T13:56:57.452198Z 0 [Note] [MY-015019] [Server] MySQL Server: Plugins Shutdown - start.
2025-01-10T13:56:57.452212Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2025-01-10T13:56:57.452230Z 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2025-01-10T13:56:57.452242Z 0 [Note] [MY-015020] [Server] MySQL Server: Plugins Shutdown - end.
[Entrypoint] ERROR: Unable to start MySQL. Please check your configuration.
[Entrypoint]

With a closer look I noticed: “[ERROR] [MY-000067] [Server] unknown variable ‘mysql_native_password=ON’.”

And indeed, in Changes in MySQL 9.0.0 (2024–07–01, Innovation Release) — Configuration Notes it is stated that the deprecated variable is now removed.

Now, my values.yaml for the mysql-innodbcluster dependency looks like:

# MySQL InnoDB Cluster values
mysql-innodbcluster:
  version: 8.4.0
  disableLookups: true
  image:
    pullSecrets:
      enabled: true
      secretName: image-private-ocir-creds
  serverInstances: 3
  routerInstances: 1
  credentials:
    root:
      user: root
      password: yes-what-would-it-be
      host: '%'
  tls:
    useSelfSigned: true
  serverConfig:
    mycnf: "mysql_native_password=ON \n"

And there it is, in the serverConfig.mycnf attribute. I guess it was added at one of our initial setups, probably based on some older recipe to create an MySQL InnoDB cluster database.

Now, I could simply remove the lines. But, what about existing users and databases then? Since it is an upgrade, obviously I have existing users. Specifically for my Gitea installation. Removing this variable potentially could break my setup.

Check the users

So, with our databases still on MySQL 8.4.0, check the users. And, since we’re running on Kubernetes, I connect to one of the mysql cluster server pods and connect to the server instance:

$ kubectl -n $NS exec pod/mysql-cluster-0 -c mysql -it -- /bin/bash
bash-5.1$ export MYSQL_PASSWORD=would-you-not-like-to-know
bash-5.1$ mysql -hmysql-cluster-0.mysql-cluster-instances.mysql.svc.cluster.local -P3306 -uroot -p$MYSQL_PASSWORD;
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 34817
Server version: 8.4.0 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Then run the following query:

mysql> select User,plugin from mysql.user;
+---------------------------+-----------------------+
| User                      | plugin                |
+---------------------------+-----------------------+
| gitea                     | caching_sha2_password |
| mysql_innodb_cluster_1000 | caching_sha2_password |
| mysql_innodb_cluster_1001 | caching_sha2_password |
| mysql_innodb_cluster_1002 | caching_sha2_password |
| mysql_innodb_cs_3e8       | caching_sha2_password |
| mysqladmin                | caching_sha2_password |
| mysqlbackup               | caching_sha2_password |
| mysqlrouter               | caching_sha2_password |
| root                      | caching_sha2_password |
| localroot                 | auth_socket           |
| mysql.infoschema          | caching_sha2_password |
| mysql.session             | caching_sha2_password |
| mysql.sys                 | caching_sha2_password |
| mysqlhealthchecker        | auth_socket           |
+---------------------------+-----------------------+
14 rows in set (0.00 sec)

Here you see that for all the users the passwords are already using the caching_sha2_password plugin.

If, in your case, a user would show up with the mysql_native_password plugin, migrate the user using:

mysql> alter user <username> identified with caching_sha2_password by <password>;

See also: caching_sha2_password as the Preferred Authentication Plugin.

Upgrade MySQL and Gitea

Now, we can update our Chart.yaml as mentioned above and the values.yaml to:

# MySQL InnoDB Cluster values
mysql-innodbcluster:
  version: 9.1.0
  disableLookups: true
  image:
    pullSecrets:
      enabled: true
      secretName: image-private-ocir-creds
  serverInstances: 3
  routerInstances: 1
  credentials:
    root:
      user: root
      password: sakila
      host: '%'
  tls:
    useSelfSigned: true

Also, we’ve upgraded the Gitea Helm chart to 10.6.0:

# Argo CD application Chart for Gitea
apiVersion: v2
name: gitea
description: An Argo CD Application Helm chart for Gitea
type: application
version: 10.6.0
appVersion: 1.22.3
dependencies:
- ...
- name: gitea
  version: 10.6.0
  repository: oci://eu-amsterdam-1.ocir.io/axufzftra6ys/helmcharts
  alias: gitea
- ...

And after a forceful update of the applications, all seems fine:

Logged on to Gitea, using the argocd user

Conclusion

Gitea 1.22.3 (Chart version 10.6.0) and MySQL 9.1.0 seem to work nicely together. But, before upgrading to this combination of versions, make sure that your MySQL users are already created by the preferred caching_sha2_password authentication plugin. And remove the mysql_native_password variable from your server config.