Step 1: Creating a Simple RPC Platform

10x Engineering Series – Microservices

This blog is a part of our series on “10x Engineering Series – Microservice”. The series covers two aspects:
— The 2-steps to lay the foundation of a microservice platform
— The amazing benefits that are unlocked
If you haven’t read the introduction, go here. Through our experiences, we hope to inspire you to build the right platforms at the right time!

Intro

The 1st step to create an amazing microservices platform is to add a wrapper / adapter layer over your service endpoints. Infact, while at it, create a simple RPC framework.

This simple first step will unlock a lot of great value (covered in other blogs of the series).

Here, we will walk you through what exactly is RPC and the simplest way to get started.

What is RPC?

RPC stands for Remote Procedure Call.

Wikipedia: In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction.

Instead of explaining, let me take you through an evolution of code into RPC. Hopefully you’ll see why it is so cool!

We will assume we are using NodeJs with ExpressJs (that is our stack, and it is fantastic!).

Level 0 — Hello World

Here is a simple server file for a “Hello World” example using ExpressJs (NodeJs).

///////////////////// server.js
var express = require(‘express’);
var app = express();
// example – hello world
app.get(‘/hello’, function(req, res) {
 res.send(“Hello World!”);
});
// example – with parameters
app.get(‘/things/:name/:id’, function(req, res) {
 res.send(‘id: ‘ + req.params.id + ‘ and name: ‘ + req.params.name);
});
// launch the server on this port
app.listen(3000);
// call => http://localhost:3000/hello
// returns => Hello World!

Problems:
Firstly: Business logic (implementation of the /hello and /things) is not abstracted to another file.
Secondly: Routing is inline and mixed with the business logic.
Thirdly: Configurations are hard coded (what do you do with multiple environments like PROD, STAGE, DEV).

Level 1 — MVC (monolith!)

Let’s abstract the business logic into a controller.js file. This file should be agnostic to the web framework or networking framework used (no req, res, app).

Let’s also split the remaining code into a server.js — that has all the initialisation of connections (port to launch at, other servers to connect to, etc); and a router.js that has all the routes and transformation of parameters/response.

///////////////////// server.js
app.listen(3000);
///////////////////// router.js
// example – hello world
app.get('/hello', function(req, res){
   res.send(controller.hello();
});
// example – with parameters
app.get('/things/:name/:id', function(req, res) {
   res.send(controller.thing(req.params.id, req.params.name);
});
///////////////////// controller.js
function hello() {
  returns "Hello World!";
}
function things(id, name) {
  returns 'id: ' + id + ' name: ' + name
}
///////////////////// client.js
const https = require('https');
https.get('https://localhost:3000/hello', (resp) => {
  let data = '';
resp.on('end', () => {
    ...
  }).on("error", (err) => {
    ...
  });
})

This still has hard-coded configs, we will come to that later.

Problems:
Callbacks.
Client to server communication is over a network. Most likely, you’d want to do async communication (NodeJs is awesome!). Because we are doing network calls, you have to deal with callbacks. Callbacks get messy really quickly — you end up nesting callbacks inside callbacks, and it gets all too confusing!

Level 2 — RPC (microservices!)

There is a simpler way to deal with all the above problems using some RPC magic!

Here, the service.js file is similar to the controller.js file. (let us move away from the MVC controller nomenclature and instead be more service driven.)

Let me not explain, just checkout how crisp it gets!

// The RPC FRAMEWORK way
///////////////////// service.js
function async hello() {
  returns "Hello World!";
}
function async things(id, name) {
  returns 'id: ' + id + ' name: ' + name;
}
///////////////////// server.js
service = require(service.js);
service_id = "hello_service"
var server = RPCFrameowork.createServer(service_id, service);
///////////////////// client.js
var client = RPCFrameowork.createClient("hello_service");
await client.things();
await client.hello();

Here, the framework:

• Automatically creates routes from the service methods
• Automatically transforms the parameters and responses
• Automatically figures the launch configuration (port, etc) from service name

All the boilerplate for server instantiation and configuration, as well as router definition is automated!

You can now focus on the actual business logic.

( How did we go from port to service_id? Read about it in Step 2. )

Async and Callbacks?

A core concept to understand communication in microservices is asynchronous calls.

Javascript has 3 ways to deal with async calls — callbacks, promises, async/await.

For RPC to imitate a regular function call, it needs to get rid of callbacks. An RPC call needs to return something that is called asynchronously. That is where Promises come in. Async/await is essentially promises with easier syntax (lets you write code as-if it were sync).

( we won’t explain promises here. Best to read up on it online if you are not familiar — it’s an important and detailed topic in itself. )

Here is an example that uses callbacks, promises and async/await. Here, suppose we had a service endpoint with this implementation:

function async echo(num) {
  return Promise.resolve(num);
}

We want to do two network calls and return the sum. So the async version of echo(1) + echo(2).

This is how it will look like:

///////////////////// client.js
//////// Callbacks in the typical way
var ret_success;
https.get('https://localhost:3000/echo/1', (resp) => {
  resp.on('end', (d) => {
    https.get('https://localhost:3000/echo/2', (resp2) => {
      resp.on('end', (d2) => {
        ret_success = d1 + d2;
      })
  })
})
//////// Promises in the RPC way
var client = RPCFrameowork.createClient("hello_service");
val ret_success = client.echo(1)
  .then(function(res) {
    echo(2)
    .then(function(res2) {
      return res + res2;
    })
  })
//////// Promises in the RPC way
var client = RPCFrameowork.createClient("hello_service");
var ret_success = await client.things() + await client.hello();

Just in 1 line! Wow!

Creating a Simple RPC Framework

Most likely, your first attempt at microservices is like ours – services are web-application servers exposing JSON APIs. Just like the examples above.

Great start!

Going from here to a simple RPC Framework is quite straightforward. Let me put down some very basic code to explain what happens underneath. (it will be over simplified here, but you’ll get the point)

Supposing you decide your services will be written as simple functions that take in params as an object, and return a Promise.

///////////////// service.js
const Service = {
  getUser: function(params) { ... },
  getProvider: function(params) { ... }
}
module.exports = Service;

Writing a very simple framework that takes your service functions and wraps them inside an http API call.

var express = require('express');
var app = express();
var service = require('service.js');
const RPCFramework = {
 createServer: function(port, service) {
   service.foreach ( method ) {
     app.post('/' + method.key, function(req, res) {
       var params = req.params;
       var ret = method.value(params);
       res.send(ret);
    })
    app.listen(port);
  })
}
module.exports = RPCFramework;

You could now extend this with a lot of interesting middleware.
You can also implement a createClient() in a similar fashion.

Service Schema: For proper implementation, you will need to give a schema for the service to create the RPC endpoints on client. You might even consider having a common schema file that all clients and the server follow.

Should you create your own or use an existing one?

We believe in using existing tech. At the same time, making our teams agnostic to the underlying choices. This means putting a wrapper around architectural choices so that we can further standardise and make it easy to use for our needs.

We created our own RPC framework in NodeJs using ExpressJs and OpenAPI specifications to define service schemas.

There are plenty frameworks out there also – gRPC/protobufs, etc. The crux is, whatever is easier for you to build and understand. Remember, the goal here is for you to be in control via a central framework. How that framework does things is details.

The beauty here is, your service implementation is now wrapped in a platform layer. You can easily change and replace implementation without affecting code for teams. This makes the teams:

• Agnostic of the networking protocol (http)
• Agnostic of the serialisation format(json/xml/protobuf)
• Agnostic of the other team’s service stack

Summary

RPC is awesome!
This blog hopefully gave you basic context on what RPC is and how to build a simple layer on top of your service as an RPC framework.

Something this basic unlocks a ton of interesting things we will come to in the other blogs.

Let’s jump to Step-2.

Next >> Step 2: Creating a Directory of Services & DBs

Sounds like fun?
If you enjoyed this blog post, please clap 👏(as many times as you like) and follow us (@UC Blogger) . Help us build a community by sharing on your favourite social networks (Twitter, LinkedIn, Facebook, etc).

You can read up more about us on our publications —
https://medium.com/uc-design
https://medium.com/uc-engineering
https://medium.com/uc-culture

https://www.urbancompany.com/blog/humans-of-urban-company/

If you are interested in finding out about opportunities, visit us at http://careers.urbancompany.com