Illustration by Darya Kozemchuk

Communication layer design for Ruby microservices

When your application grows it’s a good solution to split it into separate microservices to handle the complexity. When you decide to extract some functionality into separate service(s) two problems appear:

• How to organize communication between main application and service(s) in the way you don’t need to make lots of changes (code duplications) in different code bases.
• How to test all the parts together to be sure you don’t have “interfaces mismatch”.

This article will present an approach for creating maintainable communication layer between microservices. The code below is written in Ruby but the approach is applicable to any programming language.

“Book Reader” application

I can’t share our production code with you but in order to understand the approach, I propose you to imagine an application. It is a “Book Reader” web application where you can find books by some criteria, buy it, synchronize to your bookshelf and then read. The application is written in Ruby using Ruby on Rails framework.

Our application should be connected with lots of book providers, so there are millions of books available for users. But there is a problem. When a user tries to find a specific book we need send a search request to all the providers and collect the responses. It may take a long time. So we decided to store all available books in our own database (periodically refreshing data for each provider). Of course, it is not a good idea to mix the code and database of that storage with “Book Reader” application. Therefore we extracted this into microservice called “Book Storage”.

So, there are two Ruby on Rails applications “Book Reader” and “Book Storage” which should communicate via HTTP. These are so-called chained microservices with synchronous HTTP request/response messaging.

Let’s use word “application” to refer “Book Reader” and word “service” for “Book Storage”.

Illustration by Darya Kozemchuk

Let’s discover how we can design communication layer between these applications.

Book Storage Client

The idea is to encapsulate all the communication stuff inside separate gem — book_storage_client. And the gem is a part both of “Book Reader” and “Book Storage” applications.

Illustration by Darya Kozemchuk

This book_storage_client gem will include:

• HTTP client initialization (with authentication logic)
• Classes for initiating API calls to and from “Book Storage”
• Serialization/deserialization logic
• Mapping to “Book Storage” models

The file structure of the gem is illustrated below:

Let’s explore each part more detail.

HTTP client is built on top of rest-client library. But besides performing requests it also does two very important things.

• it creates a signature for each request and adds a few headers needed for authorizing request. Note, that Authenticator class is also a part of the gem and is used both in application and service.
• it transforms RestClient exceptions into domain specific ones.

We won’t discuss details of HTTP client implementation, it worth a separate post about authorizing and signing HTTP requests. One thing should be mentioned is that this client is used both for API calls from the main app to “Book Storage” service and for so-called hook calls in opposite direction.

API calls are performed via “fetcher” object. Take a look at BookFetcher implementation:

module BookStorageClient
  class BookFetcher < BaseFetcher
    def all
      deserialize_array(client.get(“/api/books”))
    end
  
    def find(uuid)
      deserialize(client.get(“/api/books/#{uuid}”))
    rescue BookStorageClient::Errors::ResourceNotFound
      nil
    end
   
    private
    def serializer
      BookStorageClient::BookSerializer
    end
  end
end

The “fetcher” class implements methods to get list of books and fetch one book via uuid. deserialize and deserialize_array methods are defined in BaseFetcher class:

module BookStorageClient
  class BaseFetcher
    delegate :deserialize, :deserialize_array, to: :serializer
    private
    def serializer
      raise ‘should be implemented’
    end
    def client
      Client.new(BookStorageClient::Config)
    end
  end
end

They just delegate deserialization to serializer — the core part of the gem. Each serializer knows how to serialize model and how to deserialize it. There is a code of BaseSerializer — an ancestor of all serializers:

module BookStorageClient
  class BaseSerializer < ActiveModel::Serializer
    class << self
      def serialize(object)
        serializer = new(object)
        ActiveModelSerializers::Adapter.create(serializer).to_json
      end
  
      def serialize_array(objects)
        serializer = ActiveModel::Serializer::CollectionSerializer.new(objects, serializer: self)
        ActiveModelSerializers::Adapter.create(serializer).to_json
      end
      def deserialize(data)
        deserialize_one(JSON.parse(data))
      end
      def deserialize_array(data)
        array = JSON.parse(data)[‘data’]
        array.map { |hash| deserialize_one(‘data’ => hash) }
      end
      private
      def deserialize_one(data)
        hash = ActiveModelSerializers::Deserialization.jsonapi_parse!(data)
        attrs = _attributes.map { |attr| hash[attr] }
        deserializable_model.new(*attrs)
      end
    end
  end
end

We use active_model_serializers gem for generating JSON responses.

The implementation of BookSerializer just defines attributes to be serialized and model to be built after deserialization.

module BookStorageClient
  class BookSerializer < BaseSerializer
     attributes :id, :uuid, :title, :author
     
     def self.deserializable_model
       ::BookStorageClient::Book
     end
  end
end

serialize and serialiaze_array methods are used in “Book Storage” controller for preparing response data:

def index
  render json: BookStorageClient::BookSerializer.serialize_array(@books)
end
def show
  render json: BookStorageClient::BookSerializer.serialize(@book)
end

deserialize and deserialize_array are used by “fetchers” on the client (main application) side. The “fetchers” return objects of deserializable_model class. The class is just simple struct with attributes taken from serializer:

module BookStorageClient
  Book = Struct.new(*BookStorageClient::BookSerializer._attributes)
end

So, with such simple approach, we keep all the serialization stuff in one place and can easily change data passed from service to the application.

We’ve just discussed one way of communication when the application initiates requests to service. But sometimes service needs to tell something to the application. For example, when some book is updated on “Book Storage” we need to inform the application about this. That’s why we need “hooks”. They work like fetchers but from the opposite side. For example:

module BookStorageClient
  class BookHook < BaseHook
    PATH = ‘callbacks/book_storage/books’.freeze
    
    def updated(book)
      client.post(PATH, book: serialize(book))
    end
    protected
    def serializer
      BookStorageClient::Hooks::BookSerializer
    end
  end
end

BookHook can initiate post request to the main application sending necessary data. In most cases, hooks just send minimal information about models (id or uuid of updated record), so they use their own serializers like BookStorageClient::Hooks::BookSerializer:

module BookStorageClient
  module Hooks
    class BookSerializer < BaseSerializer
      attributes :id, :uuid
   
      def self.deserializable_model
        ::BookStorageClient::Hooks::Book
      end
    end
  end
end

But one can use the same BookSerializer as fetchers do.

Testing

It’s a common problem that tests for communication layer have a lot of mocking. For our BookStorageClient, we need to mock both sides. That makes our tests almost useless because there are many potential interface mismatches we can’t test.

The solution of the problem is to write integration tests which are simulating application-service interaction both in “Book Reader” and “Book Storage” applications. There is a simple way to do it using RSpec and Puma.

The idea is to run Puma server running “Book Storage” in a separate thread and make requests to it via the BookStorageClient interface. The code below allows you to run the Puma server via RSpec tags:

RSpec.configure do |config|
   config.before(:all, type: :book_storage_client) do
     @puma_thread = Puma::Server.new(Rails.application).tap do |s|
       s.add_tcp_listener(‘localhost’, 3456)
     end.run
   end
   config.after(:all, type: :book_storage_client) do
     @puma_thread.kill
   end
end

The test for BookFetcher looks like:

describe BookStorageClient::BookFetcher, type: :book_storage_client do
  let(:fetcher) { described_class.new }
  
  describe "fetcher.find" do
    before { create(:book, uuid: ‘uuid’) }
    let(:book) { fetcher.find(‘uuid’) }
    it "return BookStorageClient::Book object" do
      expect(book).to be_a?(BookStorageClient::Book)
    end
    # — — — 
  end
end

In the main application we do the same: run the application code in separate Puma thread and use hook objects to perform requests.

That tests not only serialization/deserialization logic, but the authentication/authorization layer too. So you are absolutely sure that communication layer works correctly!

Conclusion

The discussed approach is battle-tested in our production code and developers are happy with such simple and understandable technique.

I hope the article helps you to make a final decision about splitting your monolithic application into microservices.

But with this, I leave this decision on you 🙂. In case you’re on Twitter @anton_mishchuk I’d be curious to hear your opinion.

Have a wonderful week,
Anton

Thanks for hitting the 💚 if you enjoyed this article. This will tell me to write more of it!

—
Anton Mishchuk is a Lead Ruby Developer at Matic Insurance Services .