ClanHR: booting the services

ClanHR is a new product we’re building to take care of companies’ Human Resources.

After talking about the frontend setup, this post focuses on the services layer.

Microservices

Coming from a big monolith application and having to deal with so many problems because of it, the logical choice would be microservices. However, we do have the following quote in mind:

Programmers know the benefits of everything and the tradeoffs of nothing.

— Rich Hickey

There are some people that question the start of a project with microservices. Martin Fowler’s Monolith First and Uncle Bob’s Clean Micro-service Architecture come to mind.

I don’t really know if we are building a microservices architecture. We are definitely not building a monolith. I believe that we are building a bounded context services architecture. Our aim is to try to group context and similar functionality on a service. It may not be a small or micro service, but it should have everything related to that context.

We have about 3 months of work and we currently have 2 services: directory and absences. More services are on the way (reporting, notifications, and documents).

We do have common code libraries. Instead of using microservices for shared core we are releasing specific libraries that must be used by all services. We have postgres and memory gateways, we have analytics, auth, validation and several other utilities.

We are aiming to have a set of libraries that support all services and establish a common ground for all of them. All of them should log things the same way, use the same analytics core, and the same database providers.

This brings additional difficulties. When we update a library, we need to update it’s reference on all services/libraries that use it. We do have a way to check if dependencies are up to date and use semantic versioning to know if updates are easy or might break something. But it’s a tradeoff.

Clojure

I used ruby on rails for the past three years and now our services are fully in Clojure. It’s quite a leap. It took me some time to learn functional programming and this alien language on the JVM. My objective was to write better, faster, simpler and more reliable code. I started learning Scala and the step to Clojure was soon after that.

After these couple of months I would say that it was a good bet. Clojure is very fun and challenging, and its focus on simplicity is being well embraced by the team. Being on the JVM makes it fast and reliable, and that’s something that we had as a requisite after working with ruby.

It may be hard to find manpower for Clojure. We will probably need to train people willing to learn Clojure. We don’t need to hire yet, but that may be a problem. Even so, I believe that the language is just a small part of the application. One can learn 70% ruby in 2 weeks, but a big project takes years to know.

We may have a difficult language to learn, but we’ll focus on having an easier project to learn.

Service architecture

Our services follow several of the clean architecture guidelines. We have interactors, that are responsible for performing the logic of the application.

We have controllers that adapt HTTP communication to and from the interactors. They are responsible to get the request’s raw JSON, and delivering it as clojure data to the interactors. After processing, they’re responsible for delivering the response data as JSON via HTTP.

We have gateways, that allow the interactor to communicate with other services or datastores.

The interactors are not coupled with the delivering mechanism or with the data stores. It’s indifferent for them to receive that via JSON/HTTP, XML/HTTP or via EDN/TCP. That’s the controller’s job.

Interactors can also manage data in postgresql or in memory. This translates to very fast unit tests. We take full advantage of this and on every pull request we run the full test suite for each gateway, making sure that everything is working.

We are aiming to have our interactors/services fully async using core.async. This means that all the layers interfaces interact via async channels. This will be faster and more efficient and we also believe that will make our code more readable and easy to reason about. This is not because of the use of channels alone, but because using them forces us to think more clearly on what we need to do.

Challenges

We’re still not sure on how to test service interactions (something like pact?). At this moment the services are fully tested as a unit and we have global end-to-end tests on the frontend. But we are still not comfortable with this and may try other processes.

We are still learning and our service architecture evolve every time we implement a new one. I believe that we are almost on a final template, but we do have some code that could be better and will need some refactor. It will put to the test the maintenance of a Clojure app.

We have a staging env and all the tests use that env to communicate with other services. This env is based on heroku and it’s working pretty fine. But we will need to go to production and we are already preparing docker containers for them. It’s new ground for us, and it may take some time.

There are still a lot of new techs that we need to learn and introduce: logging (logstash?), monitoring (riemann, librato), data (postgres, 0MQ, elasticsearch), docker. I’d really like to have more manpower. :)

Interested in knowing more? Follow our next steps at our medium page.

See also ClanHR: booting the frontend.