Restify.JS: Your Production Ready REST API At Scale With The Microservice Architecture.

Get up and running with the Restify.js REST API based Framework at scale in the Microservice way by building and deploying a User Microservice.

Today, microservices are taking the lead in many tech companies. However, some still build their application in a monolithic way. Their system seem to be well protected, but it may represent a security issue, where hackers can find a breach to corrupt the whole system. Therefore, services like Users Service, Payment Service, Email Service and other sensitive services residing in the same network will also be exposed.

This scenario wouldn’t have been possible if the application was split into little independent services that interact with one another through a secured API. In this case, even if the main service was corrupted, other services would still be intact and away from danger. This is an advantage of the microservice architecture.

What is a Microservice?

Microservice is compared to the Linux philosophy of small tools each doing something well, which we mix, match and combine into a larger tool. So it is a service or a tool built apart to only perform a specific task. This service can then be easily integrated into other services.

The microservice architecture adds another layer of security to your application, making it more secured and barricaded. So, in this article, we will consider a very sensitive service that almost all applications have: The User Service.

Why a User Microservice/Service?

Instead of adding a user model and a few routes and views to an existing application, this wouldn’t be practical for a real-world production application although it is totally feasible. But we care much about the high value of user identity information and the super strong need for robust and reliable user authentication. Based on this, we will develop our User Service in a microservice way. This will help build a user authentication system with the required level of security, one that is probably safe against all kind of intruders. However, a talented miscreant can still break in because, in every software program there are always some bugs here and there that the intruder could still take advantage of.

Introducing the User Service

The user service is in my humble opinion the most important service in any production application. It contains every single information about the user. Don’t just think in terms of Name, Password and Date of Birth, but consider also the user’s Bank Account Credentials, his Followers, the things he views or does daily, his Favorites Channels etc… Those information are a gold mine for hackers and other companies in need of advertising their products by targeting the right kind of user.

To limit the risk that users’ data will fall into the wrong hands, we will tightly secure and barricade the user authentication microservice. It can be used by any of our application through a well protected REST API. This will be an independent service deployed on a standalone Heroku server. Any other service/server that needs to access our User Service, must be authenticated first. It means the remote service must have the valid credentials to be authorized access to our User Service. Without any ado, let’s do this service together.

Kicking off our User Service

To make this simpler and avoid you the pain of setting up and configuring a database server, we will use SQLite as our main Database along with the Sequelize ORM. As our Node.js Framework, we will use a REST framework named Restify.js. As the main website suggests, Restify is a Node.js web service framework optimized for building semantically correct RESTful web services ready for production use at scale. This makes it perfect to build a scalable user service. We will finally deploy it on a heroku server. To start, our user service will have the below structure within a directory called users. So create a new project folder named “superapp” and inside it, create a directory entitled “users”, then create the bellow sub-directories and files within the users directory.

superapp/users/

server.js
routes/
  - index.js
controllers/
  - index.js
models/
  - users-sequelize.js
database/
  - default-sequelize.sqlite3
config/
  - default.json
scripts/
  - create-user.js
  - read-user.js
  - delete-user.js
  - list-users.js
sequelize-sqlite.yaml
.gitignore
.babelrc

server.js will contain the logic to create our user server and protect it with some credentials.

config/ directory will contain a json configuration file that represents the single source of truth for all the service routes.

routes/ and controllers/ as the titles imply are the routes and controllers of the user service.

models/ directory will contain the logic to create, read, delete and list users from the database. To do this, we will use the Sequelize ORM to interact with our database. This directory will also contain the logic to authenticate users.

database/ directory will contain our SQLite database file holding users’ data.

scripts/ directory will contain logic to test the creation, reading, deletion and listing of users.

sequelize-sqlite.yaml will be the configuration file to configure the type of database we are to use. As said earlier, we are using SQLite, but you could also use MySQL if you prefer. See Database section at the end of the article.

.gitignore is the configuration file we will use to ignore all npm modules so that they don’t move to production or a git repository.

.babelrc is the configuration file where we configure babel. Since we will be heavily using ES6+, we use babel to translate ES6 code to plain JavaScript.

package.json is the json file where we install all our dependencies and inject environment variables when the server starts. It will automatically be created when you initialize the project with npm.

To kick off the project, at the root of the project in the users directory, let’s initialize with the following commands:

npm init -y
git init

The first command will initialize the package.json file and the second will create a new git repository. That being said, let’s start installing our dependencies.

Pre-requisites:

• Node@v10 or higher
• npm@v6 or higher.

Dependencies

npm i bcryptjs config debug fs-extra gravatar js-yaml jsonwebtoken md5 restify restify-clients restify-errors sequelize sqlite3 uuid --save

Dev Dependencies

npm i babel-cli babel-preset-env babel-preset-stage-2 cross-env nodemon --save-dev

Before we go any further, let’s configure .babelrc and .gitignore. These files are to be created at the project root as shown in the project structure above.

.babelrc

{
  "presets": [
    "env",
    "stage-2"
  ]
}

.gitignore

/node_modules
/database/users-sequelize.sqlite3

We can now start creating our server, but first, we will start setting up a better way to debug our app using the debug library. The code is the following.

server.js

Next, we create authentication credentials that the user server will be expecting from any other remote service that needs access to it. So, we will create an array holding an object. We do this to ease the checking later by just iterating through the array. Therefore, let’s add the following code.

These credentials are hard-coded for now, but to add more complexity, we will need to encrypt the key later. However feel free to change the key(highly recommended) for your personal use. The next thing to do is to write the logic that will check for the required credentials.

We wrote a check function that will be passed later to Restify as a middleware. As we said earlier, any other remote server or service that needs to access the user service will first need to authenticate itself. It means that it must exactly hold the earlier credentials(user and key). This function is in charge of checking the authentication credentials before granting any access.

The first condition checks if the the authorization header and its basic Auth exist. The req object actually has an authorization header which holds the username and password in its basic Auth. This first check will save us time and resources by not bothering to look for any credentials if those objects do not even exist. But if they exist, we will then iterate through the array and check that the username and password presented by the remote server is the same as the credentials on the user server. If it is the same we set the BASIC_FOUND to true and grant access with the next() function in the chain, otherwise we deny access to the server and return a friendly debugging message “Access Denied”.

Note: Unlike Express.js and other Node.js framework, Restify expects you to always pass the next() function. As you may already know, it is a middleware that determines whether to go to the next function in the chain. In case you don’t want to go to the next function, just pass false to it as in: next(false). Nevertheless, there are also other ways to stop restify going to the next function in the chain; so you can visit Restify official website for more information, under the 5th paragraph entitled: Using next().

Now we can create a Restify server that represents the user server. So add the following code to the next line.

The createServer() is a restify method that creates a restify server. Now that it is created, we will need to configure the server with the following code.

On the first line, we enable authorization on the server using a restify plugin named authorizationParser(). The next line is passing the check function we wrote earlier as a middleware to the server we just created. This means that every single http request made to the user server will first pass through the check function for authorization.

On the next line, we enabled query string parameters with the queryParser() plugin api to be able to retrieve data in a get request through req.params. We also enabled the body object with the bodyParser() plugin api to be able to retrieve forms data sent through POST or PUT requests. This time, note that we will not be retrieving forms data though req.body but rather through req.params still. Why? Simply because, we pass to the bodyParser() a mapParams attribute set to true.

Note: In Restify, to be able to retrieve data from a get, post or any other http request, you must configure the server with plugins api to do so. There are many other plugins api available for configuring a restify server, but this is all we need for our service. However, you can visit restify official website under the plugins api section for more information.

As we would in every Node.js app, let’s handle errors with the following code.

Taking advantage of the debugger we set up at the top of the server.js file, we are listening to “uncaughtException” and “unhandledRejection” events to print error messages in case of Error and Promise Rejection. What’s next? We will now create the various routes we need to create, read, delete and list users.

User Routes

Before we create the different routes and their related controller functions, we will first configure a single source of truth for all our routes.

config/default.json

This file represents our single source of truth. The code is the following.

We can then write the logic for our very first route.

routes/index.js

Now that our first route has been put in place, let’s create its related controller in order to start the server to make sure that everything is working fine so far.

User Controllers

Let’s write our very first controller. It goes the following way.

controller/index.js

When we start our server, this route will not be of any particular use. But don’t worry, we just want to test that our user service is up and running on the appropriate port, be it in development or production. Before we start it though, let’s add to our server.js file the route we just created. We do this because we used the server object within the routes module to listen and start the service. So in server.js, import the routes module and pass to it the server the following way.

...
import usersRoutes from "./routes/";
...
usersRoutes(server);

The usersRoutes(server) function is to be called at the end of the server.js file, right after error handling functions.

To start the server, we will need to touch our package.json file by configuring the server command.

package.json

{
  ...
  "server": "cross-env DEBUG=superapp-users:* PORT=4000 nodemon --exec babel-node server.js"
  ...
}

Now, still being at the project root in our console, we launch our server.

npm run server

If it all goes well, we must see the message:

superapp-users:routes-debug User Authentication Server User-Auth-Service running at http://127.0.0.1:4000.

Our API has started taking a shape. Our main API in development will then be http://127.0.0.1:4000/users. This will be the entry-point for any of our User Service functionalities. We chose to keep it simple and stupid.

Now, using Postman, let’s test the User Service by calling the api http://127.0.0.1:4000/users. If you do not have Postman, it is totally fine, just follow along. But if you do, open it. In the url/request bar, select the GET request type, then provide the username and password authentication credentials under the Authorization tab, then send the request. Without the credentials, our request would be rejected with a 500 Error Status and a friendly response message “No Authorization Key Found”. In case, the credentials are incorrect, our request would be rejected with a 401 Error Status and a friendly response message “Server Authentication Failed”. The following screenshot well demonstrates what we mean with a successful access to the User Service.

Note: Be carefull, http://127.0.0.1:4000/users IS NOT the same as http://127.0.0.1:4000/users/. Throughout this article, the three dots(…) are to indicate the previous or next piece of code.

So, are we good? Perfect. You can now shut down the server if you want. Now let’s add our actual routes and controllers to get into the real deal.

User Routes — More Routes

Starting from here, we will be added all the required routes to create, read, delete and list users. Going back to routes/index.js, we then add the following routes.

routes/index.js

Next, we add each of the related controllers.

controllers/index.js

The register() callback function creates a new user using the create() function of the user’s model. In the latter function, you can notice we are retrieving post data through req.params insated of req.body as we said we would earlier. Since this is an asynchronous request, we use the ES7 async/await feature, to make sure the user is successfully created before we send back the response. As explained earlier, we also used next(false) at each step to tell Restify that it should not continue to the next call in the chain but to stop right there. In case you forget to call next() or next(false), even after completing the first task, Restify will go to the next function, which will result in an error. The same logic is used throughout the rest of the controller.

The next thing to do is to add a function that will find the user by email. So add the following code after the register function.

The primary role of the find() function is to look for an existing user. You are free to rename it to findByEmail() if your prefer. So before we create a user, we will first need to check if they exist. In case they do exist we return a friendly message to the user, otherwise we go on creating the user. The response sent back is an object that will determine whether to create the user or not. You will see the real use of this function when you call this api endpoint on your main platform/service. However we will be testing this soon.

Next, is to find the user by its id. So here is the following code after the above function.

The findById() is just like the find() function, except that it looks for the user by its id. The function is handy when we want to delete the user, check the user’s post, profile etc..

Let’s now authenticate the user by checking its password. The logic then goes this way.

The checkPassword() checks the user’s password when they are trying to log in. Using just the email and password, the response sent back from the user’s model is an object that determines whether the user’s credentials are correct or wrong.

It is now time to write the logic to delete a user from the database.

The destroy() function deletes the user from the database. To do this, it is necessary to know the user’s id. However, here we are deleting the user by its email. We chose to use emails because we wanted to be able to easily test later. You will see clearly in to this further.

Finally, we can write the logic to list our all registered users.

The list() function lists all users. The result send back from the user’s model will either be empty or an array of length > 1. So in case there is no user registered, we simply return an empty array, otherwise we return the list. Note that in every function, we implemented a trycatch. This helps us to handle asynchronous request errors when they arises.

So good so far, right? The last step to getting all the functionalities of our User Service up and running, is to write the logic that will actually create and save users’ data in the database, as well as performing on the database different actions such as reading, authenticating, deleting and listing.

User Model

Before we start creating the user model, let’s start configuring the database we will be using in a YAML file.

sequelize-sqlite.yaml

In this configuration, We start giving a name to our database. We then set up a password and a username to access the database. However, the most important parameters are the dialect which specifies the kind of database to use and the storage which specifies where to store the database file.

The operatorAliases parameter configures SQLite operators like equal, greater than, lesser than etc.. This simple configuration file actually helps our service to be more scalable. At anytime, we can switch to any database, be it MySQL or Postgres. In case we want to switch, we will just need to change the dialect and storage properties, rename for instance the yaml file to sequelize-mysql.yaml or sequelize-postgres.yaml and download the appropriate Node.js modules.

Now that this is set, we can start with the user model.

models/users-sequelize.js

First, we start importing Sequelize. We could have only use SQLite to write queries to interact with the database directly, but we use the Sequelize ORM to ease our development and make the user server more scalable. We are importing bcryptjs to mainly crypt users’ passwords. We could have used Node.js default file system(fs) to read a file, but it does not support promises, so we imported the fs-extra module that has all the features of the default file system but also some additional features along with promises support. Finally, we imported the js-yanl module to read the YAML configuration file we created earlier.

Then we defined the sqlize and SQUser variables. They will respectively configure Sequelize to use a database and define the User document or table if you prefer. The Op variable is defined to use Sequelize operators.

The connectDB() function contains logic to create the User table and connect to the database. The first condition checks if there is an existing database file. It there isn’t, we then create a new one by reading the YAML configuration file whose parameters are supplied to Sequelize to create the database. The second condition checks if there is already an existing User Table. If there isn’t, we create a new table, otherwise, we just return the existing table for actions to be performed upon. Anytime we want to touch the database, the connectDB() function will always get called to check for all those things.

Next, we can write functions that will interact with our database. First of all, we will start saving a new user to the database. Add the following code right after the connectDB() function.

The create() function creates a user into the SQLlite database. Sequelize has an inbuilt function called create(). As you can see, it is called upon the User table to save a new user. But before we have any interaction with the database, we must first ensure that the connection is established. For this reason we use promises to wait for the connectDB() function.

The find() function looks for a particular user in the database. This is possible through the Sequelize findOne() function that searches a unique user by a particular property, in our case: email. The if condition helps creates a new user. In case the user exists, we return an object specifying that the found property is true, therefore returns a friendly message to the user on the view. However, if the user does not exists in the database, we rather return an object specifying that the found property is false. Such information will determine whether to create the user or not.

The findById() is just the same as the above find() function, except that this time, it searches the user by its id within the database.

The checkPassword() is where we authenticate users to log them into their account. Of course, the whole mechanism is not done here. Nevertheless, we authenticate users by first checking if they actually exist in the database. If they do, it means that they have previously been registered. So the next block of code checks the email and password. Since we used bcryptjs to encrypt the password upon registration, we still use it to decrypt upon authentication. If the password does not match we return a friendly “Invalid Credentials” message to the user in the view, otherwise we sent an object that will be used to log the user in.

The destroy() function permanently deletes a user from the database. It first checks if the user exists then removes them with the Sequelize destroy() function.

The list() function as you may already know, list out all existing users in the database. The Sequelize findAll() is in charge of listing them all. We then iterate through the array returned by the Sequelize function to filter out user’s password. You may be thinking that there is no need since all passwords are encrypted. I totally understand your concern, but thinking this way would be a terrible mistake. Remember, we care much about users’ data that our main concern is to make it highly secured.

Therefore, the sanitizer() function will be responsible for filtering out the user’s password as we may notice. To finish with our model, we lastly need to export all our functions, otherwise we won’t be able to interact with the database from the controller file.

export { create, find, findById, checkPassword, destroy, list }

This last line of code ends the models/users-sequelize.js file. The final and next stage before deploying our User Service will be of course, to test it. So let’s go.

Testing the User Service

In every software development lifecycle, tests are written to ensure the application behaves the way we expect and spot bugs at very early stage. This is what we will be doing here. Note that at this stage, we are to create another server which will be a client server that handles the logic to create a new user. Create a file called create-user.js within the scripts directory as shown under the Kicking off our User Service section.

create-user.js

Using the “use-strict” mode, we first make sure that our code is safe and less error-prone. Then we import all the modules we will need to create this test. The “restify-clients” module is the library we will use to create the client server that simulates http requests against our User Service. This same client server will be used in all other tests. As you may already know, the “uuid” module will give us a unique id at the creation of every single user. Now, lets create the client server.

The createJsonClient() is the “restify-client” method that will create our client server. You need to make sure that it is listening on the same port as with the User Service. This is because, it is against this service we will be making http requests. Next we will need to provide the credentials needed, to have access to the User Service.

…
client.basicAuth(“super-admin”, “FLDBY482-KUD5-X74P-7PDZ-6W0MB46DLMN5”);

So, after creating our client server, we have access to the basicAuth() method which takes as parameters a username and a password , that are the credentials. To finish writing this test, we will write the logic that will actually create the user the following way.

To make the code cleaner and more readable, we created variables that represent some of the user’s data and a function named encryption that is responsible for encrypting the user’s password using the “bcryptjs” module. Next we create an IIFE function that will immediately get called when the client server starts. This is where the post http request will be made against the User Service to finally create our very first user. This block of code ends the create-user.js file.

You and I are all tempted by now to run those two servers to see what actually happens and make sure that our service is smoothly doing what it has been made for. But relax, we will get to this shortly. Let’s just finish with all other tests, then we can sleep on our laurels.

The next code logic is similar to the create-user.js file, except that here we will be reading the information of the user we created above. If you haven’t yet, create a file named read-user.js that contains the following code.

read-user.js

You see that the logic is quite the same and pretty straightforward using the same createJsonClient() method. However, in our get request, you may notice that we use process.argv[2]. Remember that we are using Node.js after all. In Node.js, the process.argv[] is an inbuilt API of the process module which is used to pass command line arguments. Here, we are looking for a particular argument within the command of the current running process. Therefore, the 2 within the array specifies that is the thrid argument we are looking for. So, in package.json, we will provide within the “read-user” script command, the user’s email to read the user’s info. That is how we will look for a particular user in the database.

Now, here is the rest of our code for respectively listing all users and deleting a particular user from the database. If you still haven’t yet, create a file named list-users.js and delete-user.js that respectively contains the following code.

list-users.js

delete-user.js

This is the end of our tests. Now before we start the User Service and the client server for testing the service, we will need to make some amendment in our package.json file. Open it to amend the following scripts.

package.json

Next, open two different terminals and run the following command in the first.

npm run server

This command kicks off our User Service. Keep this running. Then, run the following command in the second terminal.

npm run create-user

This command creates a new user and save it to our SQLite database. The logs shown in both consoles must indicate if the creation was successful or not. So, you can test that it was indeed created by running the “read-user” command.

npm run read-user

If successful, the logs in the console must read the information of the user whose email you provided in package.json. Do the same for the rest of the commands

npm run list-users
npm run delete-user

Now you can create as many as users you want. Just make sure that you always provide a new email in the create-user.js file. Do the same for reading and deleting the user you want by updating their email in the respective scripts configured in package.json.

Deploying to production

Now we are good to go. What remains is that we host our User Service on Heroku. The first step for doing this is to uninstall Babel from Dev Dependencies and re-install it in Dependencies.

- Dev Dependencies

npm uninstall babel-cli babel-preset-env babel-preset-stage-2 --save-dev

- Dependencies

npm i babel-cli babel-preset-env babel-preset-stage-2 --save

Since Node.js and even less Heroku do not not support ES6 codes yet, doing this will allow us to use Babel to run ES6+ codes with ease in production. In package.json, you may have already noticed the main command that will start our User Service in production:

"start": "SEQUELIZE_CONNECT=sequelize-sqlite.yaml babel-node server.js"

You can plainly see the use of babel-node. But be careful, this is not recommended. I used Babel in production for a quick fix just to have our Scalable User Service Up and Running in production. Babel is supposed to be used only in development and not in production.

However, starting from version 10+ of Node.js, ES6/ES7+ features are now supported, but not in same file extension. COMMONJS Modules use the .js file extension and do not support ES6 codes yet. But with the event of ES6 Modules, the community introduced an experimental module that understand ES6+ codes in a .mjs file extension. Therefore, I can highly suggest instead of using .js extensions, you can use .mjs extensions and write ES6+ codes without the need of Babel. Just make sure that you add in package.json the command — experimental-modules in the scripts where you are executing .mjs files. So if you decide to use this extension, your main start command could be like:

"start": "SEQUELIZE_CONNECT=sequelize-sqlite.yaml node --experimental-modules server.mjs"

This is one way of solving the problem. That being said, I leave it up to you to find another better way of running ES6+ codes in production. You can share it in a comment bellow.

Next, rename config/default.json file to config/production.json. You should also set to true the “production” value of the API’s entry-point which is in the index controller.

Now, what you need is a Heroku account. If you have none yet, then create one. Afterward, you will need the Heroku Toolbelt installed on your machine. If you do not have it on you machine, then follow this link for instructions. Once the Heroku CLI installed on your machine, open a terminal and login with your heroku credentials, providing your username and password.

heroku login -i

Once logged in your terminal, you can push the project to a git repository. Create first the project on your github or gitlab account if you haven’t yet. Then add it on your machine with the very first command:

git remote add origin https://github.com/your-username/project-name
git add -a ./
git commit -m “Created the User Service”
git push origin master

Now lets create a new heroku app.

heroku create superapp

This command creates our project on the heroku platform. Here, we named it “superapp”, but feel free to name it whatever you like. Then check if the heroku project repository was successfully added on your machine with:

git remote -v

Finally, we can push to production.

git push heroku master

This command deploys automatically the app to production. You can view the progress and activity on your heroku account dashboard. However, being at the root of the project, you can also view it from the terminal with the following command.

heroku logs --tail

We are all set. We have successfully created a Scalable User Service and shipped to production. Now, our api’s entry-point in production will be https://superapp.herokuapp.com/users. Test this api in Postman as we did earlier.

Now, do you have another application that needs to create, read and authenticate users? It could be a Photo Sharing or Booking Platform. That platform is then your first and main service. Now you have another fully functional service: the User Service. All you will need is to make both services communicate through the api we just shipped. You can also adapt this api to the need of your platform to perform whatever action you need on the User Service. Nothing is impossible, the only limit is your imagination. However, there are things we can still improve: Tests, Database and Security.

Tests

So far, we have tested our application using the restify-clients module to simulate http requests against the User Service. We’ve also tested our api’s entry-point with Postman. However, what we haven’t tested are the other remaining endpoints. So taking advantage of Postman, try to cover the rest of the endpoints. Why not also trying to write some units tests ;)

Database

After deploying to heroku, you will realize that your sqlite3 database file is being cleaned up after a while; everytime the dyno reboots. This is because the Heroku filesysten is ephemeral, thus causing you to loose all previous stored data. This is normal because, SQLite is not really a production grade database. So consider switching to MongoDB, MySQL or Postgres.

Switching to MySQL may even be easier than what you think. As we said earlier, you just need to adapt the parameters within a new YAML configuration file called sequelize-mysql.yaml, then download the required mysql module for Node.js. But note that you will have to configure a MySQL database server on heroku, which will eventually require a credit card.

Security & Beyond

We have secured our User Service as we could for now. But there is still work to be done. I will soon be releasing my next article entitled Docker: Building Scalable Applications In The Microservice Way Made Easy. With the architecture already set in place, we just developed a subnet segmentation without realizing it. This is a good design decision. Our User Service now represents a Network that we can name AuthNet, because it is an authentication system. In the Docker article, we will develop an additional subnet called FrontNet. This will be the main service that our “superapp project” will be offering. So that service could either be a Photo Sharing or Booking Platform as we mentioned a while ago. This service will then communicate with the User Service through the API we just deployed with the required level of security to have access to it. We will create a protective security wall between both subnets by dockerizing and making them communicate through a Docker Bridge Network. You can imagine each Service, independently sitting in its network performing its task. So, we will continue from there and explore together how to fully protect our superapp project at scale with this Containerization Technology, but this time on Digital Ocean. Stay safe till then.

Happy Coding!