Adding an Auditing System into a Rest API
One interesting task when developing Rest API service is adding a feature to an existing Rest API where its implementation needs to be duplicated across every Rest resources such as auditing system for all resources. This type of feature if not managed properly will increase overall application complexity and reduce code quality drastically.
This story will show you how to manage and modularize your code using Aspect Oriented Programming (AOP) technique to create a modular code that applied into multiple controllers without modifying the existing code. As an example, we will create a simple auditing system (75 lines) to track user activity and plug it into the existing Rest API created using Plumier.
If you haven’t heard yet, Plumier is a new Node.js framework designed to easily create robust and secure restful API using TypeScript. You can read the full story behind how it designed and created.
In this story we will not be using specific tools to perform AOP technique, instead, we will use Plumier middleware to do rudimentary Aspect Oriented Programming as most IoC Container libraries do. Just like the IoC Container interceptor, Plumier middleware has interception and metaprogramming capabilities which make it a perfect tool to do that.
As stated everywhere in Plumier documentation, Plumier is not an opinionated framework, meaning there is no opinion as a guide to solving a case in Plumier. So this AOP technique is not the only solution to solve a case in Plumier. You are free to use your solution that match with your requirements, Plumier just provided the building block to do that.
In this story, we will create a simple auditing system to track user activity on every request in our Restful API. Information will be saved into the database like below
Above data will be used by the administrator or auditor of the application. This data is important for further analysis of user behaviour to get what the user is mostly doing in the system for like/dislike report. This data also can be used as a simple security auditing to get information about a security breach, if a user with a specific role can access private resources etc.
Unlike application logging, auditing system requires more specific information that will be used by administrator or auditor for further analysis, for example in the above table is the
Data column, which will provide detail information about user requests such as query and body request.
When auditing a process implemented inside middleware it has access to the raw data of the body request, including data that is prohibited to the administrator or auditor to see, such as phone number, email or even user password. Saving this data directly to the database is considered a data violation. Furthermore, we need to hide or censor some part of the data so it not barely saved into the database.
Preparing The Rest API
For the existing Rest API, we will use the example code from Plumier documentation here. The provided example is a simple Rest API managed todo data, created using Plumier and Knex.js for data access. The full source code of the project can be found in this GitHub link.
To be able to save the audit data into the database, it is required to create a new domain and a database migration script.
- Create a domain model for the Audit domain at the end of the file
src/model/domain.tsDomain is a plain TypeScript class declared with some properties, in the end, it will look like this.
- Create a new migration script to add Audit table into the SQL database inside the
db/migrationsdirectory, create a new file named
1.1.0.tsand add Knex.js migration script. In the end, it will look like this.
If you want to take a preview on how the final project looks, or you are too lazy to follow the step provided in this story, you can see the project here.
Implementation Without AOP
To get a good understanding of the topic, we will have a quick peek on how auditing system usually implemented without AOP. As an example, we will add the auditing system to the
PUT /users/:id resource. I choose this resource because it’s a perfect example for auditing that requires censorship so the code covers the entire auditing system.
Before starting to add the auditing system to the controller lets take a look at the original code for comparison before the auditing system added. Below is the
UserController host the resources (some of the controller’s methods removed for clarity).
The code above looks straight forward. It only consists of two processes: hash the password using
bcrypt then save the data to the
User table. Below is how its look like after the auditing system added.
Compared with the original code, we added a big amount of code to the current logic which makes the overall controller complexity increased. Remember that the auditing system needs to be implemented in other methods handling different HTTP verbs such as
DELETE. Here is how its implemented on the delete method.
Based on two implementations above, you might notice some issues occur.
- Adding, modifying and bug fixing the auditing logic requires a lot of effort because it’s duplicated in every method.
- Testing is required for each method implemented auditing system.
- To enable or disable auditing is almost impossible without modifying the controllers.
That’s enough to make overall application complexity drastically increased.
Enter the AOP
Just like other programming tools, AOP is not a tool to solve everything. Based on its description on Wikipedia, AOP best used to modularize cross-cutting code that duplicated everywhere inside an application such as logging, global error handling, caching, etc.
Before choosing AOP to solve a programming problem, we can do some simple analysis to find out whether the new feature will cut across on existing implementations. Let’s review our previous code like pictures below.
The picture above shows the code block for the modify method. Code marked in red is code for the auditing system. Let’s compare with the code for the delete method below.
The picture above shows the auditing system for the delete method. The code block for the auditing system inside delete and modify methods are nearly the same, except that the data saved to the database is different based on the resource data and the censorship process does not occur on the delete method.
Two pictures above clearly showing that the auditing code block is a cross-cutting code that duplicated in some controller’s methods. This simple analysis is a good start to refactor duplication using AOP technique.
Refactor Duplication Into Middleware
Before we walk further into the implementation, we will have a sneak peek to the Plumier middleware for better understanding of the AOP technique. Let’s take a look at the sequence diagram below.
Plumier middleware is a sequence of invocation that runs one to another using chain of execution (named Middleware Pipeline). The invocation can be other middlewares or a controller’s method execution that handle the current request. Like previously mentioned Plumier middleware designed to have interception and metaprogramming capability which can be used to perform simple AOP technique.
Based on our previous analysis we can move the auditing process into Plumier middleware like below.
Above code showing that we move the auditing system that duplicated inside methods into middleware from line 4 through 13.
Line 4 showing that we created an audit object from the invocation context using
createAudit function, we will discuss this function later on the next section.
In line 3 we check if the current request context has
route information and has current login user information, if it doesn’t match the criteria it will proceed the next execution immediately.
The most important part is in line 6 where we call
next.proceed() to execute the next invocation. Calling this method will proceed invocation to the appropriate controller’s method handles the request. So if the request was
PUT /users/:id then the call sequence will jump from the middleware to the
modify method as pictured below.
The result of the method execution will be returned back to the middleware. It can be used for further processing, but in this example, we just returned it for the previous invocation.
The above middleware if registered in the global scope, will intercept all methods execution during all requests, that’s mean all methods will execute auditing system implicitly.
Another important part of the above middleware is creating the
audit object created by
createAudit function. This process requires more effort because the
audit data varies based on the current method executed.
Plumier has metadata information that can be accessed via request context on
Invocation object. Request context is a Koa context injected with some metadata information such as.
routeis route information contains current route handle the request including controllers and methods metadata.
parametersis parameter list in a form or array of an object that will be used to execute the method as a result of parameter binding. The array data is already sanitised and converted to an appropriate data match with the method’s parameter data type.
Based on the information above we can easily program the
createAudit function like the snippet below.
The code above shows that we perform some metaprogramming to create the
resourcedata extracted from the controller name by removing the
Controllerword on line 10 and 11.
actiondata extracted from the current HTTP Method, by translating it into appropriate words defined by
AuditActionMapobject on line 1.
parametersproperty but we need to do some censorship before it saved to the database.
Censorship function also can be done using metaprogramming. If we look closely, the censorship function is applied based on parameter data type, not to the current method. We can provide another map contains a function to censor the data like below.
The code above shows that we get the data type of the parameter of the current executed method from
context.route.action.parameters. Then we loop through the parameter values and censor the data using appropriate censor function from the map defined on line 1. The
CensorshipMap can be added later for another data type that requires censorship.
That is the last step we need to create a user activity auditing system. Next, you can register the middleware directly using the
use() method of the Plumier application in the startup code or register it using Plumier facility. In this project example, we created a
UserActivityFacility to register the middleware to make it consistent with other configuration. You can see how it registered here.
Appropriate unit testing for middleware can be added into the
test directory on the project. Plumier middleware is a stateless class so it is relatively easier to unit test. In this story, we will not create a specific unit test, but we will just execute the current integration tests that were provided from the example by using
yarn test from the command line. The execution process should automatically execute the auditing system and added some records to the
Audit table like below.
The above picture shows that all data saved properly into the
Audit table. The censorship function also works properly on the
data column. It’s mean our auditing system work like expected.
That’s it! That’s how you can add a cross-cutting feature into an existing Rest API using AOP and metaprogramming without modifying the existing code. Most importantly, think about this story before you have a plan to solve your case using AOP technique. You need to do some simple analysis if it is able to be solved by AOP and make sure not to overuse it.
Support The Project
As a final word, this story is intended to explain the power of Plumier middleware, there are more features that will be explained in the next story. Currently, Plumier is relatively new in Node.js world. The hardest part of building a framework is building and maintain a solid community. If you think Plumier can help your work and match with your need, kindly help make Plumier a better future by supporting the project on GitHub.