Communication layer design for Ruby microservices

Anton Mishchuk
May 31, 2017 · 6 min read

When your application grows it’s a good solution to split it into separate microservices to handle the complexity. When you decide to extract some functionality into separate service(s) two problems appear:

  • How to organize communication between main application and service(s) in the way you don’t need to make lots of changes (code duplications) in different code bases.
  • How to test all the parts together to be sure you don’t have “interfaces mismatch”.

This article will present an approach for creating maintainable communication layer between microservices. The code below is written in Ruby but the approach is applicable to any programming language.

“Book Reader” application

I can’t share our production code with you but in order to understand the approach, I propose you to imagine an application. It is a “Book Reader” web application where you can find books by some criteria, buy it, synchronize to your bookshelf and then read. The application is written in Ruby using Ruby on Rails framework.

Our application should be connected with lots of book providers, so there are millions of books available for users. But there is a problem. When a user tries to find a specific book we need send a search request to all the providers and collect the responses. It may take a long time. So we decided to store all available books in our own database (periodically refreshing data for each provider). Of course, it is not a good idea to mix the code and database of that storage with “Book Reader” application. Therefore we extracted this into microservice called “Book Storage”.

So, there are two Ruby on Rails applications “Book Reader” and “Book Storage” which should communicate via HTTP. These are so-called chained microservices with synchronous HTTP request/response messaging.

Let’s use word “application” to refer “Book Reader” and word “service” for “Book Storage”.

Illustration by Darya Kozemchuk

Let’s discover how we can design communication layer between these applications.

Book Storage Client

The idea is to encapsulate all the communication stuff inside separate gem — book_storage_client. And the gem is a part both of “Book Reader” and “Book Storage” applications.

Illustration by Darya Kozemchuk

This book_storage_client gem will include:

  • HTTP client initialization (with authentication logic)
  • Classes for initiating API calls to and from “Book Storage”
  • Serialization/deserialization logic
  • Mapping to “Book Storage” models

The file structure of the gem is illustrated below:

Let’s explore each part more detail.

HTTP client is built on top of rest-client library. But besides performing requests it also does two very important things.

  • it creates a signature for each request and adds a few headers needed for authorizing request. Note, that Authenticator class is also a part of the gem and is used both in application and service.
  • it transforms RestClient exceptions into domain specific ones.

We won’t discuss details of HTTP client implementation, it worth a separate post about authorizing and signing HTTP requests. One thing should be mentioned is that this client is used both for API calls from the main app to “Book Storage” service and for so-called hook calls in opposite direction.

API calls are performed via “fetcher” object. Take a look at BookFetcher implementation:

module BookStorageClient
class BookFetcher < BaseFetcher
def all
deserialize_array(client.get(“/api/books”))
end

def find(uuid)
deserialize(client.get(“/api/books/#{uuid}”))
rescue BookStorageClient::Errors::ResourceNotFound
nil
end

private
def serializer
BookStorageClient::BookSerializer
end
end
end

The “fetcher” class implements methods to get list of books and fetch one book via uuid. deserialize and deserialize_array methods are defined in BaseFetcher class:

module BookStorageClient
class BaseFetcher
delegate :deserialize, :deserialize_array, to: :serializer
private
def serializer
raise ‘should be implemented’
end
def client
Client.new(BookStorageClient::Config)
end
end
end

They just delegate deserialization to serializer — the core part of the gem. Each serializer knows how to serialize model and how to deserialize it. There is a code of BaseSerializer — an ancestor of all serializers:

module BookStorageClient
class BaseSerializer < ActiveModel::Serializer
class << self
def serialize(object)
serializer = new(object)
ActiveModelSerializers::Adapter.create(serializer).to_json
end

def serialize_array(objects)
serializer = ActiveModel::Serializer::CollectionSerializer.new(objects, serializer: self)
ActiveModelSerializers::Adapter.create(serializer).to_json
end
def deserialize(data)
deserialize_one(JSON.parse(data))
end
def deserialize_array(data)
array = JSON.parse(data)[‘data’]
array.map { |hash| deserialize_one(‘data’ => hash) }
end
private
def deserialize_one(data)
hash = ActiveModelSerializers::Deserialization.jsonapi_parse!(data)
attrs = _attributes.map { |attr| hash[attr] }
deserializable_model.new(*attrs)
end
end
end
end

We use active_model_serializers gem for generating JSON responses.

The implementation of BookSerializer just defines attributes to be serialized and model to be built after deserialization.

module BookStorageClient
class BookSerializer < BaseSerializer
attributes :id, :uuid, :title, :author

def self.deserializable_model
::BookStorageClient::Book
end
end
end

serialize and serialiaze_array methods are used in “Book Storage” controller for preparing response data:

def index
render json: BookStorageClient::BookSerializer.serialize_array(@books)
end
def show
render json: BookStorageClient::BookSerializer.serialize(@book)
end

deserialize and deserialize_array are used by “fetchers” on the client (main application) side. The “fetchers” return objects of deserializable_model class. The class is just simple struct with attributes taken from serializer:

module BookStorageClient
Book = Struct.new(*BookStorageClient::BookSerializer._attributes)
end

So, with such simple approach, we keep all the serialization stuff in one place and can easily change data passed from service to the application.

We’ve just discussed one way of communication when the application initiates requests to service. But sometimes service needs to tell something to the application. For example, when some book is updated on “Book Storage” we need to inform the application about this. That’s why we need “hooks”. They work like fetchers but from the opposite side. For example:

module BookStorageClient
class BookHook < BaseHook
PATH = ‘callbacks/book_storage/books’.freeze

def updated(book)
client.post(PATH, book: serialize(book))
end
protected
def serializer
BookStorageClient::Hooks::BookSerializer
end
end
end

BookHook can initiate post request to the main application sending necessary data. In most cases, hooks just send minimal information about models (id or uuid of updated record), so they use their own serializers like BookStorageClient::Hooks::BookSerializer:

module BookStorageClient
module Hooks
class BookSerializer < BaseSerializer
attributes :id, :uuid

def self.deserializable_model
::BookStorageClient::Hooks::Book
end
end
end
end

But one can use the same BookSerializer as fetchers do.

Testing

It’s a common problem that tests for communication layer have a lot of mocking. For our BookStorageClient, we need to mock both sides. That makes our tests almost useless because there are many potential interface mismatches we can’t test.

The solution of the problem is to write integration tests which are simulating application-service interaction both in “Book Reader” and “Book Storage” applications. There is a simple way to do it using RSpec and Puma.

The idea is to run Puma server running “Book Storage” in a separate thread and make requests to it via the BookStorageClient interface. The code below allows you to run the Puma server via RSpec tags:

RSpec.configure do |config|
config.before(:all, type: :book_storage_client) do
@puma_thread = Puma::Server.new(Rails.application).tap do |s|
s.add_tcp_listener(‘localhost’, 3456)
end.run
end
config.after(:all, type: :book_storage_client) do
@puma_thread.kill
end
end

The test for BookFetcher looks like:

describe BookStorageClient::BookFetcher, type: :book_storage_client do  let(:fetcher) { described_class.new }

describe "fetcher.find" do
before { create(:book, uuid: ‘uuid’) }
let(:book) { fetcher.find(‘uuid’) }
it "return BookStorageClient::Book object" do
expect(book).to be_a?(BookStorageClient::Book)
end
# — — —
end
end

In the main application we do the same: run the application code in separate Puma thread and use hook objects to perform requests.

That tests not only serialization/deserialization logic, but the authentication/authorization layer too. So you are absolutely sure that communication layer works correctly!

Conclusion

The discussed approach is battle-tested in our production code and developers are happy with such simple and understandable technique.

I hope the article helps you to make a final decision about splitting your monolithic application into microservices.

But with this, I leave this decision on you 🙂. In case you’re on Twitter @anton_mishchuk I’d be curious to hear your opinion.

Have a wonderful week,
Anton


Thanks for hitting the 💚 if you enjoyed this article. This will tell me to write more of it!


Anton Mishchuk is a Lead Ruby Developer at Matic Insurance Services .

Matic

A blog about our company

Anton Mishchuk

Written by

Software engineer

Matic

Matic

A blog about our company

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade