Kubernetes Operator to the rescue. How our own MongoDB Operator improved our deployments.

In this article you will learn about the challenges we faced when using a GitOps deployment and why we implemented our own Kubernetes Operator for MongoDB to simplify and speed up our deployments and reduce mistakes. If you deploy with a GitOps approach in Kubernetes and use MongoDB or AWS DocumentDB our open source MongoDB Operator may help you as well.

Photo by Stefan Steinbauer on Unsplash

GitOps, secrets, pull requests — the complexity of modern deployments

Each of our deployments is described with Kustomize as GitOps source and deployed by ArgoCD. Overlays and branches are used for staging processes and manage differences of the environments. Everything is managed via git repositories, so are secrets. We use Sealed Secrets to store them securely in our repositories at GitHub. Sealing the secrets is necessary but is hard to review, which leads to mistakes and takes a lot of time.

We execute end-to-end tests for each pull request in these GitOps repositories. Each pull request is deployed in a dedicated namespace by a temporary ArgoCD application using the same overlay as the develop deployment but from the pull request branch. Each pull request deployment is supposed to get their own databases for testing and review purposes. So we need new temporary databases and database credentials on a daily basis. Managing secrets for multiple environments causes a lot of effort and also increases the differences of development and production. That increases the risk, that deployments tested in development do not succeed in production.

Our production environments use AWS DocumentDB. Because of cost and administrative reasons we use MongoDB in development. So the automation we are looking for should be viable for MongoDB and AWS DocumentDB.

Conclusion

We should avoid to manage credentials manually. We need an automated process based on simple configuration to provide database access for our deployed services in all environments — compatible with AWS DocumentDB and MongoDB.

Requirements for a productive GitOps flow

Granting access to a dedicated database for Kubernetes workloads should be a common task in a microservice world. And yes, there is a lot of software claiming to be a MongoDB Operator for Kubernetes, like the MongoDB Community Kubernetes Operator and the MongoDB Operator by Opstree Solutions. But the main purpose of these tools is to bootstrap MongoDB Clusters in Kubernetes or like the Percona Server for MongoDB in any environment. Managing credentials and database users still requires to create your own secrets. None of the tools we found has a promoted feature to manage users in existing MongoDB clusters, which is our minimum requirement to operate on an existing managed AWS DocumentDB in our production environments.

What do we actually need to make our GitOps flow more productive?

• Create a database in an existing database instance on demand
• Create database users and their secrets for exactly their database
• Provide the credentials in the desired namespace in Kubernetes as secrets
• Minimal overhead in the deployment
• Support MongoDB and AWS DocumentDB

What we don’t need:

• Provisioning a MongoDB instance — standalone or replica set — in a Kubernetes Cluster
• Support for multiple MongoDB instances in one environment
• Knowing the credentials of database users

The solutions we found have their main focus on providing database instances. If they have support for user management, they require that the developers define the credentials.

Conclusion

We decided to build our own MongoDB Operator that focuses only on the user management of a single database instance that is provided for each environment elsewhere.

How we built our Kubernetes Operator, the technical details

Starting with the implementation, there are more obstacles to tackle and more detailed requirements to fulfill. They were not obvious in the first step but could easily be covered by our iterative development approach. But this article is not about agile software development. We will ignore the iterations here and pretend that everything was clear to us from the beginning.

First of all a Kubernetes Operator needs access to the Kubernetes API. We picked the Java Operator SDK to simplify that because most of our backend services are written in Java and we are used to the language. Setting up a basic repository with some dependencies is easy, but now we have to think about what really needs to be done.

The operator needs to react on deployed services that request a database. There are a few options, how a service could request database access. A Kubernetes deployment could be annotated with something like needs-mongodb. But that is quite indirect, not easy to document and configuration needs more and more annotations. We decided for a custom resource definition called MongoDb.

At that time, our services where configured with individual parameters for hosts, database name, username, password and so on instead of a single MongoDB Connection String. That is also related to our pull request deployments, to modify the database name automatically with Kustomize. We realized that this is not needed any more in the future and let the MongoDB Operator provide only the individual properties database name, username and password and additionally the connection string. When not using the connection string, the settings for hosts and other options must still be provided for each environment separately. We accepted this additional configuration requirement to avoid unnecessary code in the operator and encourage to refactor for using the connection string.

Services may expect different names for these properties. The custom resource definition MongoDb allows to configure how each property is provided in a Kubernetes Secret, that is created by the MongoDB Operator for the service after it created the user.

Creating users in the MongoDB instance? That requires admin access! That is a security risk! Well, MongoDB has a built-in role called userAdminAnyDatabase to manage users. That is basically admin access because a user with any privileges can be created. But you can’t execute malicious queries on data by accident. We choose this as the minimal required privileges for the MongoDB Operator.

We talked a lot about pull request deployments. Obviously, this will create many new databases every day. All these databases must be cleaned up. To be able to drop any database, the role dbAdmin is needed. That is actually too dangerous for production environments. We decided that dbAdmin should only be granted for develop environments where we deploy pull requests. While not granting dbAdmin safely prevents you from data loss in production, we want to avoid not allowed queries to the database as well. Otherwise, the MongoDB Operator needs to distinguish between technical errors and missing privileges every time it fails to clean up a database. The custom resource definition provides a configuration property to enable cleanup for each requested database when the MongoDb resource is deleted.

To avoid naming collisions and hijacking databases of other services, the MongoDB Operator will create the database name and the username from the namespace and the name of the MongoDb resource.

A database can be requested by creating a MongoDb resource like this:

apiVersion: persistence.sda-se.com/v1beta1
kind: MongoDb
metadata:
  name: some-service-name
  namespace: test-namespace
spec:
  database:
    pruneAfterDelete: true # optional, default false
    connectionStringOptions: "" # optional, defaults to the ones used by MongoDB Operator
  secret:
    databaseKey: DB_NAME # optional, default 'database'
    usernameKey: DB_USER # optional, default 'username'
    passwordKey: DB_PASS # optional, default 'password'
    connectionStringKey: DB_CONNECTION  # optional, default 'connectionString'

This will result in a database and user named test-namespace_some-service-name. A random password of 60 characters is generated for the user. A Kubernetes Secret named some-service-name with the declared properties will be created in test-namespace.

So far, we covered the requirements of the developers that want to deploy their services and security aspects:

• easy to describe request for a database
• no manual creation of secrets
• service configuration with a Kubernetes secret providing the connection string and our specific properties for database name, username and password
• automatic cleanup only in development environments
• ensure unique names to avoid privilege escalation
• data in production can only be accessed (and deleted) by the service that requested the database

It’s time to think about the process inside the operator now.

The operator needs to watch MongoDb resources that are created or deleted. On creation the operator will determine the database name and username and create the user with according privileges. Then it will create the Kubernetes Secret with the MongoDb resource as owner reference. Kubernetes will take care that the Secret is removed along with the MongoDb resource. When the MongoDb resource is deleted, the operator will delete the database user. If configured for pruning it will also drop the database. If successful, Kubernetes will delete the MongoDb resource.

Finally, it is very helpful to gain some insights about the process of the operator — especially in case of errors. That’s why the operator will also update the status sub resource and provides information about the three main steps: determining the name, creating the user and creating the secret.

Conclusion

We successfully built our first Kubernetes Operator, based on the pain we had in a GitOps world. There are still secrets to maintain, e.g. for third party APIs. But we covered the majority of recurring manual steps when deploying our services. Creating a single MongoDb resource for each service saves us so much time over creating secrets and providing databases manually. It also avoids many mistakes and reduces the differences between the environments. From here we can just focus on our main goals, simple to use for the developers and secure to use in all our environments.

As we assume that others might face a similar problem and we value open source software, we published the MongoDB Operator for the DevOps community.

The MongoDB Operator is published on GitHub and documented on GitHub Pages. We are looking forward to new users, feedback and contributions. And we hope that it is valuable for other developers that go the GitOps way with Kubernetes as well.

Thanks to Ankita Srivastava, Guido Esch, Jonas Leine and Marwen Saadaoui for the support.