Through reverse engineering and code generation to microservices

Viacheslav Tyutyunkov
Wrike TechClub
Published in
12 min readAug 18, 2022

Hello! I am Slava Tyutyunkov, a Backend Tech Lead at Wrike. In this article, we’ll detail how our backend team prepared to work with the monolith, how reverse engineering and code generation helped us with this task, what difficulties we encountered in the process, and what we got as a result.

Our system’s current state and what we aim to achieve

Wrike is a SaaS solution for collaboration and project management. The architecture of the system is a distributed monolith: one large web application with about a hundred different additional services side by side. Despite the variety of services, we cannot call the current architecture a microservice architecture. Our services work with a common database, they’re stored in a mono-repository, and most of the logic is concentrated in several large modules shared between all services.

At the same time, the monolith has many different API consumers: the main web client, mobile apps, public APIs, and integrations.

We have worked within the framework of such an architecture for quite a long time and organized its processes really well. For example, we deploy daily, updating part of or the entire system. However, such an architecture, like any, has its drawbacks, and as the company develops and grows, we have realized the monolith will tie us down. Therefore, we are gradually moving towards dividing the monolith into microservices.

We want our architecture to end up looking approximately like this:

This is not so easy to do for various reasons:

  • The product isn’t static: We constantly develop it, improve functionality, add new features, etc.
  • There are also technical aspects: Code and modules are tightly interconnected.
  • API backwards compatibility issues: For example, the Wrike mobile app has its own release cycle. However, it’s difficult for us to change something in the monolith and leave the mobile application unaffected.

We decided to first isolate the mobile API by moving it to a separate service — Backend for Frontend (BFF). In this way, we create a “facade” for the monolith and separate it from the API consumer.

BFF will allow us to focus on changes in the monolith, its structure, internal communications, and the isolation of its individual fragments into microservices. At the same time, it allows us to avoid the risk of making the API inconsistent.

By gradually implementing microservices, we want to come to this:

BFF is responsible for communication between the web application, mobile applications, and public APIs with microservices on the backend.

Getting ready to work

Preparation is an important stage, whether you want to design a new system or change an existing one. The whole team must agree on a myriad of nuances and solutions — technical, infrastructural, and organizational. In this article, we will go through the technical part since the organization of the infrastructure deserves a dedicated article.

What we decided to agree on in advance:

  • Microservices interaction protocol: We have chosen REST-Like and JSON as our data transport. We also considered other options like gRPC or RSocket, but REST suited us better. Most of the developers in the team know how to work with it, so it will be easier and more convenient in the first stages of implementation.
  • Libraries: We agreed to use Retrofit2 as a client and Jackson as a library for mapping JSON.
  • Schema description: Another problem with the monolith is that endpoints have only code and no description. This is unacceptable in the microservice world — it is impossible to build a normal interaction between microservices without an adequate API description. Based on the previous choice (REST+JSON), we chose OpenAPI for the description.
  • Process organization: Describing an API in the form of a schema is not enough. The actual interface of the service must correspond to the existing schema and control this process. We decided to use the Schema-First approach. Within the framework of this approach, the developer cannot directly change the interface and endpoint settings in the code. Everything happens through the schema: as it changes, the service interface changes. If the implementation does not match the schema, the service simply will not be assembled.

As a “foundation” for microservices, we chose a fairly standard solution — Spring Boot and Spring MVC.

Then it was all about choosing our first “victim.” Our parameters were as follows:

  • No dedicated domain
  • Highly specialized API
  • Separate release cycle
  • Special requirements for backward compatibility

As a result, we decided to create a BFF for an Android app:

  • The mobile application covers a large amount of system functionality. It has its specifics, but a single domain cannot be allocated for it.
  • The mobile application shares common endpoints with the web version, but it has its own interface design and data loading specifics. Having an interface that is specialized and optimized for a mobile application is an important factor, and BFF can help here.
  • The monolith is deployed daily, and a mobile application cannot afford this. On top of that, the user migration to newer versions is quite slow.
  • We need to keep endpoints backwards compatible for a long time. Currently, the Android team follows an agreement to maintain backward compatibility for at least six months.

Step one: Create a BFF service

First, let’s look at a part of the app-to-monolith interaction between the mobile application and the monolith via BFF.

We created an empty service, set it up, and launched it:

Everything worked. The service is up, there is no traffic yet, but we have passed the first stage.

Step two: So where do we get the data?

We are building a BFF as a “facade” for the monolith. In this case, it is logical to draw data from the monolith. We do this using its current API.

Choose a method: There are several ways to get data. The first is to use a custom client. If the client allows you to receive data through REST or internal communications with the monolith, you can use it. In our situation, there was such a client, but it was not suitable for our tasks. We have a public API, but this was not enough considering the requirements of the Android application. In particular, there are differences in the data model and presentation — the Android application focuses on internal specifics that are not available via the public API.

We decided to think of the monolith as a large microservice and tried to integrate it into the overall architecture. To do this, we needed to create a schema and describe the monolith in general terms.

If the company already has a description of the system schema, you can use it. In our case, each team describes the front-end interaction protocol in its own way, and there is no general schema. We needed to create one.

Create a schema: If the schema is needed for a small sub-domain or subset of endpoints in a monolith, you can describe it manually. Updating might be an issue in the future, but in general, it is possible. We did not want to manually write a schema for 150 endpoints, so this method did not suit us.

Another option is to use a ready-made solution. For example, if the endpoints are described via Spring MVC, the springdoc-openapi library allows you to get the schema using the annotations available. But we use a custom web framework, so this method was not for us either.

We decided to write the necessary library ourselves. This is where reverse engineering came in handy. We analyzed the endpoints — most of them look similar to each other.

Input (in our case, a separate class that describes the model), output (what we give to the client), and some meta-information in annotations or configs.

This structure fits easily into the OpenAPI schema:

The schema looks standard and uncomplicated. We wrote a library that generates a schema based on the structure of endpoints and aggregated a complete list — about 150. Then we processed all the endpoints and got a schema. We published the schema to artifactory to reuse it in BFF. This is also necessary so that the front-end can take the schema to get a declarative client on its side.

We have described the monolith schema, and BFF now sees it almost like a microservice. Large, not the most convenient, but you can make it work.

Step three: From schema to code

Next, it’s time for code generation. With the help of the schema, we provide BFF with a declarative client. To get data from the monolith, we processed the schema through the OpenAPI Generator and added the features needed in our case.

We got all the necessary models according to the schema:

We got a declarative Retrofit2 client:

This allowed us to “declare” the client in the service, use it, and stop thinking about whether we are working with a monolith or a microservice.

Step four: Proxying

Initially, we decided to stick to the current protocol and only change the endpoint to minimize the labor costs for the mobile development team. In our case, BFF turned out to be a proxy with a hint of the monolith’s specifics.

We have described the BFF schema by reusing the components we got in the previous step:

Next, we generated the interfaces for Spring MVC. To minimize the number of errors in communication between microservices, we see that developers do not independently describe endpoint interfaces in microservices. The Schema-First approach postulates that only the schema is described, and the interfaces are generated automatically. It reduces the risk of developer error in the implementation and ensures consistency in microservices interaction.

We generate models in the same way.

We get proxy requests using the Retrofit2 client:

The handler logic is described as follows:

  • Take input data from the request
  • Add the necessary headers
  • Make a request to the monolith using the resulting declarative client
  • Wrap the received data and give it back to the client

We have completed the last step in the diagram by implementing the interaction between the mobile client and the monolith via BFF.

That’s exactly what we deployed in the first version, and it worked. True, there were some problems with proxying and collecting metrics, but that’s what pilot projects are for — discovering weak spots and fixing them.

Analyzing what happened

It seems that everything is fine, and we can continue to produce microservices. But let’s take a closer look at the endpoint proxy code in BFF, the very code that allows you to receive data.

In essence, it is a request to the monolith through the use of common architecture. Microservices work just the same.

Let’s take another look at the implementation of request proxying:

If you take a closer look at the code, you can see there are a lot of monolith endpoint specifics. We need to pass an authorization token: When a request is sent either between microservices or from BFF to the main system, we need to explicitly pass authorization headers. It is also necessary to transfer additional data for routing between different system segments to work correctly. In our case, this data is the user account ID.

Because we chose REST, the processing code is mostly boilerplate. We can get the data and work with it only when we ensure the request is sent and returned with the correct status.

If we use a similar approach for each call inside microservices, it will not make our lives much easier. That is why we decided to see what could be optimized in the code.

What we wanted to change:

  1. Eliminate boilerplate code.
  2. Do not pass or fill in the general parameters.
  3. Distance from protocol and data mapping. We wanted to describe the microservice as a regular bean instead of tying interfaces to specific REST implementations (Retrofit 2) and remove mapping (Jackson) from the models’ description.
  4. Leave the opportunity to operate at a “lower” level (streaming, finer status processing, etc.).

To solve these problems, we resorted to code generation once again.

Fine-tuning code generation

We divided client generation into two layers/stages.

The first layer is the microservice interface. Only the service API is generated at this level. No annotations, no mention of Retrofit or anything else.

Services

We did the same with the data model and generated a DTO from a schema with builders. This way, we completely isolate the system from implementation.

Models

The models haven’t changed, we just removed the annotations.

The second layer is the implementation of “transport.” This step generates a declarative Retrofit2 client and Jackson mixins for the models (needed to connect the real model with how data is transported over the network). As a default implementation of the microservice interface, we added calls to the Retrofit2 client and moved the entire processing boilerplate to this level.

The Retrofit2 client looks similar to the previous version — we just added an additional wrapper for method responses to remove part of the logic from the service.

Retrofit2 client

Retrofit2 is an additional wrapper that implements part of the logic, in particular, handling the response status. This wrapper can be moved to a separate library, which is what we plan to do in the future. Now the wrapper is generated according to the template and located next to the rest of the classes.

Using the given client, we prepare a standard implementation of the service. On the one hand, we give the developer the opportunity to use a ready-made template — there are the necessary calls and processing to operate only with top-level models. On the other hand, this service contains all the necessary dependencies to implement the corresponding method manually. This, for example, can be useful in case you need to process the response in a special way or organize a different way to call a remote service.

Mixins are Jackson technique to add descriptions to models without changing their code.

To make the whole schema work with minimal effort, we prepared the default configuration for Jackson, Retrofit2, etc. The developer will only have to connect the configuration to the project.

Registering a service bean using the default implementation with the Retrofit2 client:

We are currently using Retrofit2, but this approach allows us to switch it for another tool — a custom http framework, Spring OpenFeign, or something else.

What was the result?

  1. We removed http/rest processing boilerplate code.
  2. We separated the model and interface of the service.
  3. The handler code in BFF is clean and readable.
  4. We use the approach for other microservices.

The mobile development team is now working on “extracting” one of the microservices from the monolith. To do this, we describe the service’s schema, generate an interface according to this schema, and substitute the local beans we already have.

Now we are teaching the system to operate via a specific interface. When we are ready to take the database and service code out separately, we will only need to replace the transport with http, and everything should work.

BFF is in production, the second service is on its way, and we are almost done with the migration.

Conclusions

Choosing an approach is an important step in redesigning a system. What technologies will be used, how all elements of the system are connected — all of this should be agreed upon in advance. We spent lots of time at this step and had to review different problem-solving options, but it was worth it.

Reverse engineering is a good way to describe the current system. Studying the current code, processing it, and obtaining a schema allows you to describe the system and solve the problem of automation.

Code generation simplifies life and eliminates boilerplate code. Code generation helps to solve many problems and allows you to unify the developers’ approach to the implementation process.

This approach allows us to gradually transition from a monolith to microservices.

I’ll be glad to answer your questions and comments!

--

--