Cloud-Native Configuration Management by GitOps #2
To begin with, one of the most critical topics of an application environment is the configurations. Moreover, every app has at least two environments: test & prod. For example, a basic application needs at least four or five configurations. When you multiply these you got 10 config params.
By the same token, for an enterprise application, you need additional test environments like stage & QA. At the same time, you’ll have at least 10 or 20 apps per development team. When we multiply these 5 (environments) * 5 (config params) * 20 (applications) -> at least 500 configuration key and values!!! I won’t mention about “think that you are Netflix” :)
You will have to deal with a huge amount of configurations and settings. How can you manage all of these?
Twelve-Factor App
To be able to build modern, scalable, maintainable software-as-a-service apps, you need to externalize your configurations (one of the twelve factors). Besides, by externalizing configurations, you can deploy your apps into different environments without modification. There are a couple of ways that you can achieve this. With this intention, I will mention about the best practices that we are using in Trendyol. You can check-out the twelve-factor app details here.
Spring Cloud Config
In Spring Boot applications, it is very handy to use the Spring Cloud Config. Simply, you only push your configurations in a GIT repository. When a config changes, there is no need to redeploy an app; just make a POST request to the actuator endpoint. In detail, you can follow this tutorial from Baeldung.
Implementation
- Add below dependencies to the server project
- Add below dependencies to the client project
- Add below configurations to the server project’s application.properties file
- Initialize a GIT repository as configuration storage under the configured URL
- Add below configurations to the client project’s bootstrap.properties file
- That’s all! Your API is ready to consume properties from the configuration server. For example, you can use properties like below
Pros & Cons
- Ease of management: The configs only stay in a single repository.
- Same properties hierarchy: The hierarchy structure inside the application.yml can be the same in the config repo.
- No need to redeploy/restart: The applications won’t need to redeploy/restart when config changes.
- The need for an extra service: You need to maintain an extra service (spring cloud config service).
- Single point of failure: If your Spring Cloud Config service is down, you won’t be able to deploy apps or update configs.
- An extra dependency: The Spring Boot applications must contain spring-cloud-starter-config dependency.
- Bean refreshing: Spring Cloud Config refreshes the beans manually and this can cause problems on a running API.
Kubernetes Environment Variables
Correspondingly, managing configs with environment variables is a widely used technique. Notably, with the power of Kubernetes and Spring Boot, it is very easy.
Specifically, you can override application properties via environment variables in Spring Boot. You just need to set the property keys in the Screaming Snake Case format. For example, for the property named “app.url.user-api” you can set the environment variable with the name “APP_URL_USER_API”. The environment variable will automatically override the value.
Besides, we store configuration values in a separate configuration repository. In addition, we have applied a clean pipeline to deploy those configs in Gitlab. Thus, we can boot-up a fresh Kubernetes cluster very fast when faced with any problems.
As mentioned above, it is very easy to set environment variables to Kubernetes.
- Set config variables to config maps (or secrets).
- Configure deployment.yml file with the environment variables section.
In Trendyol/Marketplace team we decided to use this configuration management with Gitlab. Thus, we can use powerful predefined Gitlab environment variables like CI_PROJECT_NAME.
Implementation
- To begin with, we have separated configurations into three categories: configs, secrets, and toggles.
- As seen above, we created configs and toggles with config maps and secret values with secrets. On the other hand, CI_PROJECT_NAME is an environment variable set by Gitlab. Thus, it is the name of the project repository.
- After setting values to Kubernetes, the second step is the apply those configs to the deployment.yml file.
Note: You can apply all values inside a config map once to the deployment.yml file with the keyword “configMapRef”. For more information pelase follow this url.
- Hence, check out how we used the CI_PROJECT_NAME. Previously, we have set it at the config YAML files. Then, we have used the same parameter at the deployment.yml to match the predefined configs and secrets in Kubernetes.
- Next, create a separate project to deploy configs. For this, in Gitlab, we created a subgroup named “cfg” and created configuration projects inside here for each base project.
- Last but not least, here is a sample of our config project pipeline in Gitlab:
- After running the pipeline of the configuration file, you can deploy your app with the env variables we have set above.
Pros & Cons
- A clean management of configurations per environment.
- No need for additional dependency or service.
- No need for additional technology, just using OS environments.
- Applications must restart when config changes.
Spring Cloud Kubernetes
The Spring Cloud Kubernetes is a mix of Kubernetes environment variables and Spring Cloud Config. While you are using the environment variables, you can apply configuration changes with a basic POST request.
In Trendyol, the PIM team has used this practice. For the details of the implementation and more, please head over to this post.
Pros & Cons
- A clean management of configurations per environment.
- No need to restart applications.
- You still need extra service.
- Single point of failure. You won’t be able to deploy applications when the config server is down.
Trunk-Based Branching Model
With starting to use Gitlab, we also switched to the trunk-based development model.
Above all, this development model provides agility to the development team. Hence, changes and new features can easily and quickly deploy.
To explain, the trunk-based development model depends on a single GIT branch (master). Thus, developers push their developments only to the master branch. To be able to do this, the merged changes should not break anything in production. So you should test your code well and the project must have very low complexity.
To be able to use this model safely, the CI/CD operations must be powerful and well. Also, your team should provide a high quality of testing mechanisms. Lastly, we have decreased the complexity of our project by migrating to Gitlab while externalizing the configurations.
In trunk-based development, your changes should not affect production in an undesired way. So, if you have developed a future, it is not finished yet and you have to test it on the test environment (merge it), you should release it with feature toggles. As mentioned before, we have separated toggles into a different config map in kubernetes.
Also, as described before, you can easily apply those toggle values to your deployment.yml file.
As a result of this, a developer should add and deploy new configs with the configuration project and then add the toggle values to the deployment file before release. It’s that simple.
TL;DR
In Trendyol, we apply 12-factor-app methodology. Therefore we have externalized the configurations.
We use three different solutions for this: Spring Cloud Config, Kubernetes Environment Variables, Spring Cloud Kubernetes. With this intention, we can apply different solutions to different problems.
Spring Cloud Config is very easy to use and simple to manage. But, the config server creates a single point of failure.
Kubernetes environment variables are very handy. As a result of this, you don’t need any additional services or technology. In reality, sometimes it can be very complicated when there are lots of config params. Also, you need to create separate repositories per application to bootstrap the configurations.
In addition, Spring Cloud Kubernetes is a mix of these two. Thus, it contains the pros and cons of both. For example, you won’t need to restart the app when config changes. But, it can be hard to manage the properties in environment variables. Last but not least, the config server creates a single point of failure.
Conclusion
On the whole, it is very handy to externalize configurations. Hence, you can run your APIs in different environments without modification. In that case, there are a lot of solutions to apply. On the other hand, you should apply proper solutions to the proper problems.
Finally, please explain how you manage configurations in your development environment. Lastly, please ask your questions if any or share your knowledge by writing in the comments section.
Personal Links
- Twitter: https://twitter.com/RequestParam
- Website: https://www.btasdemir.com
