What I was doing wrong — managing micro-services common dependencies

Series of my personal discoveries, design concerns, time waste activities, patterns inapplicabilities, and other issues shared during the team regular retrospectives.

I believe we are not unique here that micro-services adoption (as a company strategic goal) is not only about independently deployable units. Polyglot development, two-pizza team, better agility, and overall teams independence, faster time-to-market, and smaller damage impact when a failure occurs makes this paradigm attractive despite increased implementation costs.

But there is always expectation, and there is reality.

After starting adopting microservices as a core design principle we experienced the next challenges:

Team independence

One more controversial statement is that microservices teams are self-sufficient and independent. Probably yes for a very small percentage of use-cases when produced service has no other dependencies or these dependencies are relatively static and do not imply support activity.

If your services have more than just computation layer abstraction, but persistent, security, observability, synchronization, then most likely you want to minimize code duplication and increase code reuse across different components of a big system. So you need to move commonly reusable components and services in shared libraries. Another reason for using shared code — consistency across different sub-systems. Having contract defined and shared across all actors reduces the possibility of broken communication channels because of unrecognized input data contracts.

As a product team, we also started experiencing the need to keep shared code in a separate library to speed up the development of related data services.

How it was

Per classic dependencies management flow we used the shared library as an independent project with its own release life-cycle. This can be represented in the next diagram:

classic dependencies release flow

This flow is good enough as allows:

• manage artifacts independently and link into any service as a dependency from a remote trusted repository, e.g.

dependencies {
    compile 'com.company.shared:some-component-starter'
}

• support collaborative model (like OSS), so it can be the single component owner who respects the contribution to this project based on a variety of use cases
• independent release process ensures that deployed artifact passed a certain level of quality requirements (static code analysis, unit tests, release notes updates, etc)

But after the adoption of this release flow for all supported dependencies (about a dozen of different sizes and responsibilities), we detected that there is a specific threshold in delivered code complexity when flow becomes very inefficient.

Developing a feature in a vacuum

Extending common DTO with additional fields and implementing new service behavior in a vacuum — are different implementation complexity tasks.

With complex features, you have a chance to get into the loop:

iterative development

1. code increment in the shared library
2. release snapshot version
3. verify in dependant service
4. if verification is not passed — goto “1”

As each artifact upgrade attempt most likely requires an additional code review process and CI execution time — we can just imagine how much extra time is needed to produce a complete artifact by following this process of dependency management. When a single feature spans through 2 or more dependencies in deep — complexity and time waste activities increase exponentially.

Also usually feature is developing for a specific service to solve specific business needs now and later just reused by other products after some pitching activity. So in reality this process looks next — you build a specific feature in the data service, and after the verification cycle moves it to correspond shared library, clean it up in the corresponding data service, and then activating back by linking a shared library. And again the high probability of an adoption loop here.

Making a holistic view on feature adoption and shared library use, also taking into account retrospective complaints about the structural complexity of such approach, so we made the decision to rework the dependency management process to make it more efficient.

Sub-project vs sub-module

If your team is the main contributor to the shared library and your services are hardly dependent on it, so emerged a question — is this module really “shared” from the perspective of its use, or is it a “core” module for your services and “shared” to another dependant services? Who is the main beneficiary of the adoption new version of the shared library?

Reviewing all cases when the shared library was used by other teams we finally understood that the majority of all cases — an attempt to share specific features that were developed for specific services to be used by other services with zero implementation and adoption cost. In other words — our libraries are like the market with different business helpers and features that customers (services) can activate: metrics to expose domain characteristics, REST API interceptors to adopt multitenancy, Vault sidecar integration, and many more. At the same time, each of these features was a core for a specific service and supported by a specific team.

So we integrated such modules as sub-projects, e.g.

dependencies {
    compile project(':some-component-starter')
}

where “some-component-starter” is a project that is linked by relative path, so the root project can access its files like from the relative folder, at the same time we can be sure that the sub-project will be built first and its code and jar will be added to the classpath.

This makes life much easier as you can develop features in a sub-project and use it immediately in root service with no artifact promotions. And most important — you can release updated shared library artifacts once all verification is passed in the main service release CI.

Release many artifacts once CI passed

So in updated CI, we added an extra “release artifact” step that is responsible for building and releasing all sub-projects as independent artifacts once all verifications are passed from the service that uses it. And only when passed, so new artifacts will not be published until the dependent service doesn't acknowledge successful adoption. As well we check that particular sub-project has any code changes to avoid empty releases when hitting this CI step.

If shared code is part of many services and there is no single (primary) support team for it — we continue to use a separate repository for this module and then use “git submodules” to combine both approaches in a more effective manner: separate repository and direct link as sub-project into service-owner.