Microservice Monitoring; node-xyz and RisingStack Trace

In this article, we will discuss how node-xyz can be integrated to work with RisingStack Trace.

What is node-xyz?

node-xyz is microservice microframework for node. It is a low level , minimal, yet comprehensive toolkit that can be used to easily develop and deploy microservices. node-xyz consists of two main components:

• xyz-core: which is the microframework that we will be using to write our services.
• xyz-cli which is the command line tool that we will be using to deploy our services. xyz-cli can be configured to work with Trace very easily.

This tutorial is using xyz-core v.0.4 and xyz-cli v0.4.1. Since node-xyz is being heavily developed at the time, there is the possibility of minor changes in the future.

Writing the microservices

In this tutorial, we will not focus deeply on the details of microservices since it can get pretty complicated and dependent on your application domain. Instead, we will create mock tasks with different types (CPU and IO tasks) that represent real tasks and deploy them as our microservices.

We will create 2 service types:

• a Front service: This node will receive HTTP requests, translates them to internal messages and dispatches the messages.
• a Worker service: which exposes the two tasks explained above.
• a Client service: this node will represent internal clients that might need to use Worker's services. In other words, Client is another node that might expose functionality to users, but it is also consuming Worker's service.

The services will be deployed to port 4000, 6000 and 5000 respectively. We will also launch a series of attacks to our Front node to represent external clients.

I will explain some of the details of xyz-core in this tutorial, but I highly recommend reading xyz's Getting Started document to get more familiar before going any further.

Let’s start writing the microservices:

Worker Service

Worker Service

Even for this small example, it’s worth mentioning that placing an IO intensive and a CPU intensive thask in one process is not a good idea. Nonetheless, the aim of this tutorial isn’t to teach you about microservices. It’s about xyz and Trace

Client Service

This node is a bit simpler since we will not .register any services. Instead, we will use .call() to send messages to Worker's tasks.

Client Service

Note that:

• For simplicity, all messages are empty and don’t have any payload.
• No need to say, the path of ther service, like /task/io, is its identifier and should be used by a caller to reach to correct callee.
• As you might’ve seen, none of the nodes are informed about the IP:PORT of one another. They use a seed node to join the system. This procedure is a part of xyz's Ping mechanism which helps services discover and explore each other.

At this point, we can have a small test. We have created two services, namely client.ms@127.0.0.1:5000 and worker.ms@127.0.0.1:6000 and we can run them both to see them working. For now, we will not use xyz-cli and run them with node command.

Start with:

You will see iterative logs that indicate task/cpu and task/io are not found. This is fine since we haven't launched worker.ms yet!

Next, in a new terminal:

After a few seconds you will see:

Which indicates that the two services are now synced. Rest of the logs should show things like:

Let’s write the last service, Front.

Front Service

A front service should usually accept requests from external clients and translate them into internal messages. This process can be quite complicated depending on the business logic of the entire system. Yet again, we will use just a simple form of translation for simplicity. In the front service, an Express server will accept only post requests with /service?service_path=SERVICE_PATH url. It will also send responses to the external clients since all messages are HTTP and we simply can do that (note that this wouldn't have been easily possible with UDP or Async messaging). Let's see the code:

If you run $ node front.js, and send a request to it like:

You will get

Which is totally fine since front hasn't connected to other nodes using its seed node. If fact, you can see how front has responded locally to the message:

Which is logged after Front fails to execute:

Notes about all three kinds of services:

• Internal messages are sent via Client instance while external message are gone through Front
• Note that all nodes use their selfConf.host as their global identifier, so be sure to replace that with a valid static and remote IP if you are testing this code in a VPS.
• All nodes will have a selfConf.seed instead of having systemConf.nodes to sync and discover other services. This is because we want to deploy more instances of each service later.
• While Sync message passing is fine for external clients, usually it is preferable to use an Async messaging for internal messages. In this tutorial, we will use HTTP Sync message for all communications for simplicity. xyz provides some Async messaging mechanisms that you can use.

Wrapping it up with xyz-cli

You can now run the three nodes individually and test them. They should work just fine. But we want to do it easier, that is why we will use xyz-cli. You can install this module globally using:

One of the commands in xyz-cli is dev. This command will run a bunch of microservices according to a configuration file. This config file is pretty simple and self-explanatory:

Create a file named xyz.json:

Notes:

• CLI will override ports if you provide them. In this case, we are giving them the same values that they had in selfConf.transport.0.port. You can read this page to learn more about how xyz overrides configurations such as ports.
• each object in nodes[] accepts more options, but we are not filling them now for simplicity.
• stdio indicates the destination of each process's stdout and stderr. By default, it's console which is not good because the logs of all nodes will be printed in one terminal. If you use file value for this key, like now, a new folder named log will be created and each process' log will be written to a separate file. This is much more reasonable.

with this file in the root of all of your files (since our services are simple now we use a flat file structure):

you can run:

After this, a list of all nodes should be printed:

xyz-cli is an interactive command line tool, meaning that you can keep entering commands to get more info and do more stuff, like killing a node, creating a new one and… You can further investigate your nodes with commands such as top:

the message rates given make sense because the front node is basically idle at the moment, and the client is sending messages to worker (~13msg/sec).

Wondering what Ping Interval is? See this page.

You might notice that all nodes, regardless of their stdio status, will start writing their outputs to file/terminal only after they have been initialized. This is a clean approach, yet, it kinda makes it hard to catch exceptions at runtime, like things as simple as a typo. You can always add a -e to xyz dev command if you sense that something is wrong. This flag will cause all nodes to output their logs to the current terminal during initialization phase, which is very critical. So if any of them have an exception, you'll see it.

We can talk for another hour about xyz-core and xyz-cli, but as you might recall from the title of this article, this wasn’t our main goal! Let’s keep things simple like this and switch this project to be monitored by Trace.

PS. In case you actually want to read for another twenty minutes about xyz you can read these two articles on Medium.

Monitoring with Trace

While it’s just about… ok to see a list of nodes deployed with xyz-cli, we can’t ignore the fact that it’s a basic tool. If you want to deploy multiple nodes across multiple servers, xyz-cli will lack some functionalities. To fulfill this requirement, Trance is a perfect solution and can be integrated with xyz-cli with no hassle.

Adding trance Configuration

In order to get started with trace, you can sign up for a 14 day trial. This will be just enough to get us started with this service. Next, you should read the Getting Started section. As mentioned, there are two ways to integrate a node process with Trace:

• using environment variables
• using a config file.

While both are possible, using the former solution is much easier. So we will choose that.

After signing up with Trace and creating an infrastructure (which I’ve named node-xyz-test), you should create services. For this tutorial, we will create 3 services, concomitant with the three services that we have.

Per each service, you will receive a set of instructions to follow in order to integrate the process with Trace. Each service will get a unique name and an apiKey. The apiKey is common among all services in an infrastructure.

Next, we must include Trace module in all of our services. You must only add the following line at the beginning of each file, given that you have installed @risingstack/trace:

Finally, you should add the service name and apiKey to each process at runtime. Hopefully, xyz-cli provides a handy way to do this. You can just add one key, named env, to each node in xyz.js:

You must only replace {YOUR_API_KEY} inside the file.

Reminder: If you want do deploy nodes across multiple servers, you must replace selfConf.host in each node with the remote address values

And there it is! Out sweet microservices appear inside Trace:

Services should appear in your panel:

The Topology should also start to shape:

With message rates being added within a second:

Also, don’t forget to see how the Metrics page to see some important information.

Notes:

• During this tutorial, I deployed all services in a single VPS and ran a benchmark test on them using Apache Benchmark, aka. ab. That’s why you see an external client sending messages to Front. The benchmark was similar to this:

ab -k -p post.data -c 10 -n 20000 HTTP://SERVER_IP:4001/service\?service_path\=task/cpu

post.data file is irrelevant since we are not using the post body but the query instead.

Remember that Front node was listening for external clients on port 4001.

• Aside from the thick lines between
• Client -> Worker
• External -> Front | Front -> Worker

which were our job, what are the other thinner lines indicating small message rates (~10rmp)? what about the circulating lines from one node to itself? Those are because of the Default Ping mechanism in xyz. This mechanism basically keeps track of all nodes inside a system and what functions they are exposing, hence it need to send some messages under the hood every once in a while to check other nodes (this mechanism does not distinguish local or remote nodes and sends iterative messages to check all of them, including itself). This is why you can simply call ms.call({service: ...}) and the message will be redirected automatically, even if the destination is in another host all the way across the globe.

• During the peak of response time in the last image, I was taking heavy benchmarks from ?service_path\=task/io, which is more intensive.

You can read the full documentation of xyz and Trace if you want to learn more!

The full code of this tutorial can be obtained from here.

I hope that you have enjoyed this article. I will sone publish the second part of this article, in which we will scale out our worker service and use an internal load balancer to distribute the messages among them.

In the meantime, any comments or suggestions are welcomed.

Originally published at gist.github.com.