How OneShop managed complex configurations with multitenant microservices architecture

In Deutsche Telekom Digital Labs, the OneShop team arise a problem to manage complex configurations in the e-commerce multitenant ecosystem. OneShop platform currently has 18+ microservices that are deployed over AWS EKS K8. As we are building this product for multiple countries, there are always many environments to deal with. Along with the on-boarding of any new Telekom country (We call it NatCos), handling of these environment configurations was becoming messier day by day.

Well, detailed information about the OneShop platform, you can find it here.

To give you practical examples let’s Imagine you have 10 microservice which has multiple environments, e.g. development, testing, marketing, UAT, production, and maybe more. Every environment has its own configurations to manage. Alas! The count does not stop here as this kind of environment set up has to be separate for all countries. Microservices deployments and releases should have different configurations to manage for each NatCos.

Now let’s talk about the challenges we faced with such an ecosystem.

1. What is the release or build version deployed on each service?

2. What set of configurations running in each service?

3. How quickly can I configure the configurations such as timeout properties, URL properties, cache properties, and many more without redeploying/restarting?

4. How to debug the application by changing the log level without restarting?

5. How to debug the application by taking thread dump/heap dump hassle-free?

To solve the above-mentioned problems, we came across a few solutions

Database approach: You can keep the configurations directly into the database. This will solve the configuration problem but building a full-fledge feature with audits like git and good UI to support is always time-consuming. Also, this was not fulfilling the other customized requirements mentioned above.

Spring Cloud Consul Config: We did a POC on Spring cloud consul config, but this does not fit in our use-case. In this approach, it does not sync consul KV to any git-based repository. We were more inclined towards git-based configuration management.

So Finally, we manage to do a successful POC over Spring Boot Admin. Of course, with some customizations! :-)

Why we choose Spring Boot Admin (aka OneShop Service Admin)?

1. As OneShop microservices are powered by the Spring boot framework, so it was easy to plugin this.

2. Ready-made UI already in place.

3. It helps to view/update properties on runtime.

4. It provides a feature to play with the log level.

5. It provides the capability to easily access heap dump and thread dump.

Now we decided to move ahead with the Spring Boot Admin, but our problem does not end here.🙁

1. Spring framework provides features to update any property/bean in run time which we call refreshable bean/properties. But in our services, we were using feign clients to make network calls, and they are not refreshable.
2. We did not want to introduce an additional dependency in each microservices to register themselves with spring boot admin so that we don’t add more dependencies and configurations because of this.
3. How to secure Spring Boot Admin to limit the features based on roles and responsibilities and to avoid any misuse?.

How we engineered this solution to stitch our use-cases and making it useful across OneShop services:

1. To solve the feign refresh problem, we decided to take this challenge and played with the feign, spring framework, and proxies to make the feign refreshable. I know you will be keen to know how we achieved this but hold your nerves till we publish this 😊
2. To solve the additional dependencies/configurations problem, K8s (Kubernetes cluster) came to the rescue. As our services were deployed in Kubernetes, we took the help of the same, so that Spring Boot Admin auto-discovers all the running services.
3. To solve the permission problem, we made use of Key Cloak authorization, which was already being used for other purposes, we integrated the key cloak in SBA and made its operations role-based.

Centralized view of some of the applications in SBA.

SBA Applications

Hey, I am a hardcore developer, talk in terms of some code snippets😠 .

Kubernetes Configurations:

1. Kubernetes service rest API feature should be enabled.

2. Service account should be created for the pod in which SBA service is running and required roles need to be provided.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: sba-readonly
rules:
resources:
- services
- pods
- configmaps
- endpoints
verbs:
- get
- watch
- list

3. As all the services running inside Kubernetes may not be the right candidates for discovery, hence filtering is required to filter the required one. Add some labels in the Kubernetes service to make them eligible for discovery.

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "shoppingcart",
    "namespace": "dev",
    "labels": {
      "management.context-path": "actuator",
      "sbascan": true,
      "appName": "shoppingcart",
      "namespace": "dev"
    }
  }
}

Microservice configurations:

1. Enable all required management endpoints.

SBA configurations:

1. Dependencies involved:

<dependency>
 <groupId>de.codecentric</groupId>
     <artifactId>spring-boot-admin-starter-server</artifactId>
     <version>2.2.2</version>
 </dependency>
 <dependency>
     <groupId>org.keycloak</groupId>
     <artifactId>keycloak-admin-client</artifactId>
     <version>4.8.3.Final</version>
 </dependency>
 <dependency>
     <groupId>org.keycloak</groupId>
     <artifactId>keycloak-spring-boot-starter</artifactId>
 </dependency>
 <dependency>
     <groupId>org.springframework.cloud</groupId>
     <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
 </dependency>
 <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-security</artifactId>
 </dependency>

2. Add label filter property to discover only eligible services,

spring:
  cloud:
    kubernetes:
      discovery:
        service-labels:
           - sbascan: true

SBA Application Detail View

With just a few configurations and customization, we were able to make it work for us. Indeed, It was a very interesting problem to solve. I hope this article helped in some of the tech learnings! :-)