Fast-tracking Integrations using Code Generation

One of the most common features developed in almost all multi-vendor e-commerce systems is vendor/supplier integration. But the drawback is that most times, each vendor’s APIs(both functionally and technologically) may or may not work with the suppliers’ current system design.

So at Agoda, we created a composable system with clearly encapsulated modules where we can easily plug in new vendors. This platform consumes vendor APIs and normalizes them for Agoda systems using feature APIs. So, if we need to integrate a new vendor, we add that integration to our platform, make a few simple configuration changes, and that vendor is ready to be onboarded.

How Agoda connects to Vendors

How We Onboard New Vendors

There are four steps in the vendor onboarding process: Discovery, blueprint, implement, and test. Not to mention that a single integration can be broken down into multiple feature-level integrations. Apart from the discovery step, the rest of the process can be done in parallel for each of them.

process of integration

The Problems We Faced

We were integrating with various flight vendors, and one of the major problems we faced was that these flight companies are not tech companies(in the core sense); hence their tech infrastructure was not up to date.

Another issue we had was language barriers. Most of these vendors are from geographically diverse locations, so performing the blueprint mapping was quite a hassle. So we began considering a better long-term solution that would either eliminate this step or, if not, at least reduce the work for us.

Implementation, can it be optimized?

Now let’s talk a bit more about the Implementation part. This is what the internal architecture of the Vendor Integration platform looks like:

The modular architecture of vendor integration platform

From the diagram above, the whole architecture is modular, with each vendor having an independent connector module that encapsulates all the entities required to connect to the vendor.

• The architecture is very similar to Onion architecture. The infrastructure and service layer is wholly decoupled from vendor modules through contracts.
• Vendor modules are self-contained and are deployable in themselves if need be (example: one vendor has too much traffic and requires dedicated servers).

Vendor module exposes features through common interfaces.

According to Agoda Systems, each vendor has the same normalized features exposed to the infrastructure layer through interfaces. This is what a common search feature would contain:

Search feature with its components

Since most vendors had to provide the same features, we decided to unify the architecture of vendor modules by copying common skeleton code per vendor and customizing it according to their needs.

This process helped us in two ways:

1. No design brainstorming is required for new vendors: Due to the unification of the vendor module design, new vendor modules can now plug into the existing architecture and use the same design. Because of these design unifications, new team members had no trouble integrating new suppliers.
2. The intent of templatization: We decided to develop our architectural skeleton at the beginning of the implementation phase to save time and effort since we noticed that it is being reused and has essentially become boilerplate code for us.

Vendor templates with Giter8

First, we created a simple template for a whole integration module that includes a boilerplate and provides blanks for vendor-specific implementations. To do this templatization, we used giter8. This is how it works:

Here is what a search service file template would look like

And once we render this file using Giter8 for a vendor whose name, for example, isSomeRandomVendor, it would look like this:

Similarly, we will generate mappers, validators, HTTP/SOAP clients, vendor data models, etc.

Entities which are vendor-independent are fully generable from the get-go while others can only be partially generated. Implementation can be completed per vendor logic. Here is a breakdown of the two kinds of entities:

Fully generable and partially generable entities

Limitations of Giter8

While giter8 was useful at the beginning, it lacked some features. It had no support for iterative generation of code. By iterative generation, we mean rendering the same template file/code snippet multiple times with different template values. Because of this limitation, we could not do the following:

• Generate vendor network clients properly: Each vendor had a different number of APIs. For each API, there was a method in the network client to connect to it. Because we cannot vary the number of methods in the template file and cannot do iterative generation in runtime, we could not generate proper network clients.
• Generate multiple vendor calls in a single Agoda feature call: Not all vendors are the same. We might have to follow different processes to do the same thing on two different vendors. A common use case is shown in the image below.

Variations among Vendors

For vendor 1, we do a single search call, but for vendor 2, we need to call auth before calling Search. This would require the vendor one search service to look very different from vendor two search service (with multiple client calls). Also, there’d be more mappers, validators, etc., for vendor two since it’s doing two calls to the vendor. But because the giter8 template is fixed and static, it does not support that. So, to address this, we explored a few alternative tools and finally settled for ScalaMeta!

ScalaMeta, and how powerful it is!

ScalaMeta, used alongside giter8, proved very effective for generating code using a template. We broke down the generation into two parts:

1. Generate from template: Generate a basic integration module using the giter8 template.
2. Transform using ScalaMeta: Break down the generated code into syntax trees and use custom-written transformers to mutate those syntax trees according to vendor config.

These transformers do quasiquotes substitution in the syntax trees specified in the config. There are only a fixed number of variations that we face among vendors (like the number of Vendor APIs in network clients, mapping of Vendor APIs to Agoda APIs, etc.), and we implemented transformers for each of them.

The vendor config required for them can be found early in the Discovery step of our vendor integration process.

With ScalaMeta, we only need to supply the vendor configuration previously stated, and we can start creating all the boilerplate. So highlighting the same use case that giter8 wasn’t able to cover:

• Vendor 1 config for a generation:

Vendor 1 config, which calls Search directly

• Vendor 2 config for a generation:

Vendor 2 config, which calls Auth before Search

Additionally, we can adjust changes in our service during the generation with this configuration change. After that, we need to populate vendor-specific mappings and their connection config, and our integration is ready to use!! Some vendors have use cases that might need to be handled separately, but they will not be counted in the boilerplate.

Moreover, ScalaMeta offers room for experimenting with and expanding the generation framework.

Some other things to note

• We added a format step at the end of our generation script, so it’s automatically formatted after a module is generated.
• To ensure that no new changes are breaking our generation framework, we created a build that generates a demo vendor module and makes sure it is compilable. We made that build a part of our CI/CD pipeline, so no changes go ahead unless that build is passing.

Conclusion

At the time of writing this article, with a single run of generation script (which takes around one min.), we’re generating close to 100 files with around 2 KLOC (varies with config provided). To put this into perspective, a completely done vendor integration usually has around 150 files with about 4 to 8 KLOC.

Comparing the effort required to integrate two different vendors is tricky as those vendors might have poles opposite tech capabilities. Still, the effort needed for integration decreased by around 50% because of the code generation framework.

Acknowledgements

Significant thanks to Kenny for contributing to this framework. Lots of appreciation to Bartek and Hywel for always encouraging us to experiment. And kudos to the awesome Connectivity team for always having our back.