Swagger, NodeJS, & TypeScript : TSOA
The following exposition is part of a series of articles concerning, Practical Web Development and Architecture. This article continues to focus on the theme, ‘separation of concerns’, demonstrating how the concept can be applied to a NodeJS server application development using the example ws-node-demo. The development of the front-end application ws-ngx-login-demo, a consumer of the NodeJs application is explained in the the article Optimal Angular : PubSub With NGRx. Both front-end and back-end applications are bundled as a FullStack integrated development environment using Docker, which is delineated in the article Docker is my {I.D.E}.
The ws-node-demo application reviewed in the article was written in TypeScript, utilizing the Express framework to establish a web server infrastructure. It supports User authentication with JWT (jsonwebtokens ) and communicates with a Mongo database by way of the Mongoose object modeling services. Documentation for the REST API is created with tsoa, a tool providing scaffolding of code and generation of Swagger JSON for showing RESTful documentation of the application with Swagger UI tools.
I developed the back-end with NodeJS and a Mongo data store, using a series of Docker containers with Docker Compose. If you are unfamiliar with Docker or want to run the application without Docker, you can do so by cloning it and building with standard NPM commands found in the package.json file. However, you must install Mongo on your client machine in order for the application to properly function. If you roll with Docker, then grab the entire ecosystem from here.
Why use TypeScript ?
Using TypeScript to develop a NodeJS application provides the same type of code base across the entire stack. There are many nuances and ‘out of the norm’ steps needed to develop JavaScript in the classical object-oriented manner offered in languages such as Java , Python, C++, Visual Basic .NET, Ruby, and ActionScript. TypeScript provides “strongly typed” declarations, which provide better precision and performance than writing in JavaScript. As s a superset of JavaScript, TypeScript must be compiled in order to be loaded as JavaScript in the browser and/or the NodeJS ecosystem.
One important concept to remain cognizant of when developing with TypeScript for browser and/or NodeJs, is how 3rd party libraries are imported. NPM library packages needed in a typical NodeJS application can often be located by creating a search term that proceeds the name of the package with ‘@types/’, as in ‘@types/express’ on the NPM site. Many of the most popular NPM packages written in JavaScript like Express can be imported in the package.json’s dependencies listing, which should also be included in the devDependencies.
Remember, the final distributed version of the NodeJs application will be consumed as Javascript not TypeScript.
"dependencies": {
"express": "^4.14.0”,
...
}Added to the package.json devDependencies
"devDependencies": {
"@types/express": "^4.0.30",
...
}With the advent of the DefinitelyTyped library, the number of developers using TypeScript has gained momentum over the past year, and thus, the DefinitelyTyped repo, which anyone can contribute to0, contains hundreds if not thousands of directories that are used to wrap existing libraries written in JavaScript. The @types/express package used in the application can be found here (https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/express)
Itinerary
- Development Set Up: Summarize the development tools and configuration.
- App Directory Configuration: Comprehensive description of the contents and approach to the structure of the application directories.
- Processing Data: Brief summation on the code pattern and how it can support optimal developer output.
- Document Generation: using ‘tsoa’ to scaffold the application and generate documentation.
Development Set Up
Swagger, a part of the Open API Initiative, provides a set of tools to describe and document a RESTful api. In theory, the idea of code generation sounds great, but thus far I continue to find it somewhat lacking. While the generation of code may not be a fruitful endeavor, the documentation, Swagger UI, produced from the Swagger Schema ( swagger-api.yaml/swagger-api.json) is an invaluable resource.
Over the last several years there have been a number of open-source projects created to unify the orchestration Swagger documentation and code scaffolding, such as swagger-tools. Like other initiatives of this sort, the process always begins and ends with the proficiency of the Swagger schema created from the outset of the project. Quite frequently, I find, as the nuances of the Swagger schema become more complex, accurate code generation becomes hampered if not impossible.
Thus, I was pleasantly surprised to discover tsoa, created by Luke Autry. Instead of first creating a Swagger Schema in order to generate the code scaffolding, tsoa reverses the process by generating the final Swagger Schema from the code created. Some code generation or updating does take place, but the majority of the code is created by the developer. This process provides the most valuable asset: sound documentation and testing.
App Directory Configuration
The directory structure of the application emphasizes an architectural approach based on a “separation of concerns”:
src/data-layer/ : responsible for organizing how the data will be stored and accessed.
- adapters : implements the set up of Mongoose for connecting to a Mongo database.
- data-abstracts : delineates both the Schemas representing the structure of each Mongo Collection and the Documents representing each set of data in the Collection.
- data-agents : implements the query transactions against the data store for each particular collection
- model : contains a Typescript class representing the definition of the data portrayed by the Document.
src/business-layer/ : implementation of business logic and other resources need by the service-layer and/or middleware
- security : contains apparatus for creating tokens and processing security checks on specific API request.
- validators : contains schema and processing logic for validating data sent with API request.
src/service-layer/ : contains the processes for establishing API endpoints in the form of routes which will facilitate responses to data request.
- controllers : serve as the basis for processing data request associated with routes. The custom controllers (User and Authorization) extend tsoa controllers, using decorators to associate router endpoints to specific functions exposed in each of the controllers. These functions, commonly referred to as C.R.U.D. (create, read, update, and delete) implement the basic GET, POST, PUT, PATCH, and DELETE processes for transacting with RESTful api.
- request : contains TypeScript interfaces representing the attributes that make up each of the various types of request.
- response : contains TypeScript interfaces representing the attributes that make up various types of responses.
src/presentation-layer/ : contains the presentational views offered by the application.
- documentation : contains the Swagger-UI files and directories facilitating the display and mechanisms for making REST based API calls and presenting the responses.
*Since the responsibility of this NodeJs application is essentially providing data against request, there are no other views but documentation.
src/middleware/ : contains resources to establish the server configuration as well as a place to store utility processes shared across the application.
- common : currently contains the instantiation of the ‘logger’ that can be shared across the application
- server-config : contains vendor specific implementation of the node server framework vendor, ‘Express’ as well as the all important ‘routes’ configuration organizing the REST API endpoints.
A brief explanation on configuration files
src/server.ts : initializes our web server with our chosen NodeJs framework, Express.
package.json : serves as the manifest in all NodeJs application, specifically delineating external libraries required for building the application in the ‘dependencies’ and ‘devDependencies’ section and providing a ‘scripts’ section provides commands necessary for building, running, and packaging.
src/types.ts : As previously mentioned, package.json external libraries should accurately portray what will be included in both development and production modes. However, it may be the case that a TypeScript definition does not exist for a desired JavaScript package. In such cases, the types.ts file can be used to declare libraries we would like to use as is the case with ‘express-winston’ used with in the logging apparatus.
tsconfig.json : provides options for TypeScript when it performs the task of compiling to JavaScript.
tslint.json : provides a listing of rules that can be applied against the code to maintain formatting standards.
src/server-config: contains the core infrastructure for instantiating the web-server.
tsoa.json : establishes the various locations of files needed to generate the swagger.json, which will eventually be used to by the Swagger UI to represent the API documentation. A couple of key configurations are
- listing the Security definitions which will be used to signify protected data access
- hosting and domain configurations, the directory path to place the output of our TypeScript compilation, as well as the label ‘/api’ used to signify the handling and processing of request data rather than other kinds of resources.
- the all important route configuration is use to identify the file used to stand up the web-server application, the location of the directory where the routes.ts can be found, and finally where the file(s) reside used in conjunction with our security apparatus are located.
Processing Data:
The primary responsibility of most web-servers is to accept and process request from clients; to this end the data flow in our application request reflects this purpose.
1 API request to GET a specific User based on the id of the User is sent to the server:
http://localhost:8080/api/Users/591cc1bc2b9622b151c9a68aThe routes.ts file located in the middleware directory captures the request base on the URI ‘/api/Users/:userId’
//middleware/routes.ts
....
app.get('/api/Users/:userId', authenticateMiddleware('api_key' ),
function(request: any, response: any, next: any) {
const args = {
userId: { "in": "path",
"name": "userId",
"required": true,
"typeName": "string"
},
authentication: {
"in": "header",
"name": "x-access-token",
"required": true, "typeName": "string"
}
};
let validatedArgs: any[] = [];
try {
validatedArgs = getValidatedArgs(args, request);
} catch (err) {
return next(err);
}
const controller = new UsersController();
const promise = controller.GetUserById.apply(
controller, validatedArgs); let statusCode = undefined; if (controller instanceof Controller) {
statusCode = (controller as Controller).getStatus();
}promiseHandler(promise, statusCode, response, next);
});
....function authenticateMiddleware(name: string, scopes: string[] = []) {
return (request: any, response: any, next: any) => {
expressAuthentication(request, name, scopes).then(
(user: any) => {
set(request, 'user', user);
next();
}).catch((error: any) => {
response.status(401);
next(error)
});
}
}function promiseHandler(promise: any, statusCode: any, response: any, next: any) {
return promise
.then((data: any) => {
...
})
.catch((error: any) => next(error));
}
function getValidatedArgs(args: any, request: any): any[] {
return Object.keys(args).map(key => {
const name = args[key].name;
switch (args[key].in) {
case 'request':
return request;
case 'query':
return ValidateParam(args[key],
request.query[name],
models, name)
....
}
2 Since access of the data provided by this URI demands a User to have been Authenticated (logged in and allocated a token), the function authenticateMiddleware(..)..is executed first
function authenticateMiddleware(name: string, scopes: string[] = []) {
return (request: any, response: any, next: any) => {
expressAuthentication(request, name, scopes) ..
...
}
}calling the expressAuthentication() method which resides in the business-layer/security/Authentication.ts file. Using the new “async” capabilities of NodeJs, the method is established as Promise chain and will thus return a Promise indicating whether the User has a valid token.
await authService.getAuthById(payload.userId);While the token can be verified without the use of a Promise, in order to ensure the User Id is valid and is associated with a logged in User, a request must be sent to the data store, which does require a Promise.
import { AuthServiceCheck} from '../../data-layer/data-agents/UserDataAgent';
import { IUserDocument } from '../../data-layer/data-abstracts/repositories/user/IUserDocument'
import { verifyToken } from './token-helpers'
let authService = new AuthServiceCheck();async function expressAuthentication(request: express.Request, securityName: string, scopes?: string[]): Promise<any> {
const token = request.headers['x-access-token'];
....
const payload = verifyToken(token);
....
let authResult = await authService.getAuthById(payload.userId);
if(authResult && !(authResult instanceof Error)){
...
return Promise.resolve({ authorizedUser:true });
}
...
return Promise.reject( new Error('jwt token malformed'));
}
};
3 The UserDataAgent.ts located in the data-layer/data-agents provides all transactional access the data-store for storing and processing User based data.
async getUserById( userId:string):Promise<any>{
...
return result
}export class AuthServiceCheck {
userDataAgent = new UserDataAgent();
async getAuthById(userId:string):Promise<any>{
return await this.userDataAgent.getUserById(userId);
}
}
4 If the User does not have a valid token or the User’s id can not be validated, the request will no longer progress through the rest of the application and a response indicating an error due to the lack of authenticity will be returned. However, if the authentication is valid, the request will continue to progress through the routes.ts, by next establishing an ‘args’ Object delineating the structure of the parameters that must accompany the URI request. This ‘args’ object will be verified by the ‘getValidatedArgs(…)’ method, which basically wraps ‘ValidateParam(…)’ : a function exposed by the ‘tsoa’ library.
function getValidatedArgs(args: any, request: any): any[] {
return Object.keys(args).map(key => {
const name = args[key].name;
switch (args[key].in) {
case 'request':
return request;
case 'query':
return ValidateParam(args[key], request.query[name], models, name) ...
}
ValidateParam method, essentially matches the ‘args’ object against the parameters delineated in the swagger.json . If the parameters are deemed invalid, progress ceases and an error response is generated. When the parameters have been deemed valid, the request continues on to a controller (UserController.ts) in the service-layer.
....
try {
validatedArgs = getValidatedArgs(args, request);
} catch (err) {
return next(err);
}const controller = new UsersController();
const promise = controller.GetUserById.apply(controller, validatedArgs);
....
5 The controller, in this case UserController.ts, calls the UserDataAgent.ts’s method ‘getUserById(… )’ in the data-layer in order to retrieve data from the data-store (Mongo) associated with this particular User Id.
@Security('api_key')
@Get('{userId}')
public async GetUserById(userId: string, @Header('x-access-token') authentication: string ): Promise<IUserResponse> {
let result = await this.userDataAgent.getUserById(userId);
if( result && result.username){
var aUser = new UserModel(result);
return <IUserResponse>(aUser.getClientUserModel());
}else{
if(result){
throw result;
}else{
throw{
thrown:true,
status: 404,
message: 'no such user exist'
}
}
}
}Controllers in the service-layer not only provide access to the data-layer through Agents used for negotiating queries against the data-store, but controllers also can provide point of access to the business-layer to process business rules. While not used in the above method, one such case can be reviewed in the UserController.ts method RegisterNewUser()
@Post()
public async RegisterNewUser(@Body() request: IUserCreateRequest): Promise<IUserResponse> {let vaildationErrors:any[] = await
validateUserRegistration(request);
.....
}
Although the initial validation of necessary parameters is performed in the routes.ts implementation, a second round of validation is used to ensure the data defining the parameters for a new User passes specific standards. The ‘request’ parameter, which contains the data defining a new User, is processed by validateUserRegistration(…) method found in the UserValidation Processor.ts implementation located in the business-layer/validators directory.
async function validateUserRegistration(userReqObj:any): Promise<any>{
let validUserRegData = new UserValidationSchema(userReqObj);
var regex = new RegExp('^[A-Za-z0-9$]+$');
let validationResults = await validate(validUserRegData);
const badPW = regex.test(userReqObj.password);
let constraints =[]
if(validationResults && validationResults.length > 0 || !badPW){
if(!badPW){
constraints.push({
constraints: {isAlphanumeric: "password must contain only
letters and numbers and $"
},
property:"password" });
}
forEach(validationResults, (item)=>{
constraints.push(pick(item, 'constraints', 'property'));
});
}
return constraints;
}6As previously described in step 3, the UserDataAgents.ts method returns the results of the query on the User collection to find the User with the id initially supplied in the URI api request. Unlike the previous invocation of this method during the authentication process, we are looking for User ‘id’ supplied as a parameter in the request rather than the User ‘id’ of the entity making the request.
7 After completing the transaction with the data-store in the UserDataAgent.ts, a Promise in the form of a vanilla object is returned to the UserController.ts indicating failure or success. When a successful retrieval of a User with the given ‘id’ is achieved, the data associated with said User is constructed as a UserModel that will be resolved as an IUserResponse Promise to the routes.ts.
@Security('api_key')
@Get('{userId}')
public async GetUserById(....){if( result && result.username){
var aUser = new UserModel(result);
return <IUserResponse>(aUser.getClientUserModel());
}else{....
If some mishap occurs in the processing of the query in the UserDataAgent.ts, such as a broken connection to the data-store, a result without the intrinsic ‘username’ will be returned which indicates an Error should be returned as the response. A similar Error response is created when the supplied ‘id’ does not exist for any User in the data-store. In either case, an Error will be thrown.
public async GetUserById(....){if( result && result.username){
...
} if(result){
throw result;
}else{
throw{
thrown:true,
status: 404,
message: 'no such user exist'
}
}
}....
8 Whether the UserController.ts result of processing the original request throws an Error or it is is successful, the routes.ts will handle the result in the promiseHandler(…) function.
app.get('/api/Users/:userId',
authenticateMiddleware('api_key'
),
function(...){const promise = controller.GetUserById.apply(controller,
validatedArgs);
...promiseHandler(promise, statusCode, response, next);function promiseHandler(promise: any, statusCode: any, response: any, next: any) {
return promise
.then((data: any) => {
//handle successful responses
if (data) {
response.json(data);
response.status(statusCode || 200);
} else {
response.status(statusCode || 204);
response.end();
}
})
//process thrown errors
.catch((error: any) => next(error));
}
9 If the response was successful, it is returned and no further process is implemented. However, if an Error exist and was thrown, the next(error) in the catch block pass the response to clientErrorHandler(…) established in the server.ts as part of the node frame work initializing process.
function clientErrorHandler (err, req, res, next) {
if (err.hasOwnProperty('thrown') && err.thrown) {
res.status(err.status).send({ error: err.message})
} else {
next(err)
}
}
nodeFrameWork.app.use(clientErrorHandler);This completes an example of how data flows through application and is processed base on a RESTful api request. Inherent in many back-end web server applications, is the inevitable cross-cutting.
Cross-cutting concerns are parts of a program that rely on or must affect many other parts of the system. They form the basis for the development of aspects. Such cross-cutting concerns do not fit cleanly into object-oriented programming or procedural programming.
Such is the case with the authentication process in the business-layer that deviates from strictly accessing the data-layer from the service-layer. Perhaps, the routes.ts file may be suited for the service-layer, but as it seems to be a core structure in standing the server up it also makes sense for it location in the middleware directory. While delineation of a “separation of concerns” is not as emphatic, as might be achieved in a browser client application, a focus on specific processes is still plausible through the various layers and facilitates a more intuitive approach as the features of the application expand.
Document Generation
Creating the documentation of the RESTful api produced by the swagger.json and presented in the Swagger UI tool is automated through using the ‘npm run’ commands in the package.json .
To run the entire process:
$ npm run startThis will invoke package.json “start” command in the ‘script’ section
"start": "npm-run-all -s clean build start:simple",1st it wilrun the “clean” scripts:
"clean": "npm-run-all -p clean:*",
"clean:dist": "rimraf dist",
"clean:cov": "rimraf coverage",2nd it will run the “build’ scripts,
"build": "npm-run-all -s build:swagger",
"build:swagger": "npm run swagger-gen && npm run routes-gen && tsc && npm run copy-swagger-ui",
...
"swagger-gen": "tsoa swagger",
"routes-gen": "tsoa routes",
"copy-swagger-ui": "ncp ./src/swagger-ui ./dist/swagger-ui",- creating the swagger.json from the ‘swagger-gen’ command,
- invoke the ‘routes-gen’ command to reappraise the routes.ts file ensuring it has been configured correctly and the RESTful endpoint associations with particular controllers is valid.
- the ‘tsc’ command will compile the TypeScript files into Javascript files using the “outputDirectory”: “./dist” attribute in the tsoa.json file to identify where resulting JavaScript files should be placed.
- Finally the SwaggerUI directory implementing the client documentation portal, which resides in the presentation-layer/documentation, will be copied to the ‘dist’ directory that was created in the previous ‘tsc’ command.
The Swagger UI client can be accessed at ‘http:localhost:8080/docs’, which was previously established in the tsoa.json file.
//tsoa.json
"outputDirectory": "./dist",
"entryFile": "./src/server.ts",
"host": "localhost:8080",
"basePath": "/api"Once the page is Swagger UI can be loaded in the browser, a listing of t the available API commands can be viewed by clicking on the ‘default’ link. The result is presented below.
To test the validity of an API request, select an api request such as POST /Users. The window should appear as below. In the request text box, the following object should process correctly as a valid user
{
"username": "taosing",
"password": "password",
"firstname": "fifty",
"lastname": "cents",
"email": "zip-zap@ez.com",
"admin": false,
"isLoggedIn": false
}
While running the single command is the easoest path toward correctly building and structuring the application, it is useful to review the process on the ‘tsoa’ GitHub page. As is the case in many development efforts, the tools rarely produce seamless results. It may be necessary to run tsoa commands individually in the beginning to understand exactly what is produced.
$ npm run swagger-gen$ npm run routes-en
Hopefully, this article has provided insight regarding how a “separation of concerns” can be achieved in a NodeJS application. Thin Controllers in a service-layer relying on references to the business-layer and data-layer for implementing process demonstrates how such a structure can eliminate code duplication and enable the sharing of services between controllers. I also demonstrated the value of tsoa and how easily this tool facilitates a use of the Swagger UI for generating documentation is accomplished.
