From Class Diagram to Node.js API Server with Loopback.io

As a Developer Advocate for IBM Cloud, I often teach students, early startups and developers, how to go from idea to Proof of Concept (PoC) using Design Thinking and Agile or IBM Garage Method as IBM likes to call it. One of the critical steps in that process is to go from a Class Diagram to implement an API server with persistent storage. By far the best framework I have seen to do this is Loopback.io, one of the many great open source projects IBM as one of the biggest drivers of Node.js has contributed to the Node.js ecosystem.

I often use a specific demo application in many of my tutorials and talks called QAVideos. QAVideos is a technical FAQ application with screen-recorded Questions and Answers. Think of QAVideos as the love child between Stack Overflow and Youtube. At the end of the Design Thinking sprint, you should have a prototype, either a lo-fi prototype or a hi-fi prototype. I developed my prototype for QAVideos using MarvelApp.com. The screens or wireframes for QAVideos are shown here.

To view the QAVideos prototype in action go to the MarvelApp.com/QAVideos project here.

After the prototype is done, I always first create a UML Class Diagram. The use case diagrams for instance are a natural representation of user stories in Design Thinking, but they also are great input for creating scenarios for test features in Test Driven Design (TDD) in Agile. The class diagrams are great input for creating database schemas, data models and REST APIs.

With Loopback.io you can generate a Node.js REST API server, the API documentation and UI, the data model and database schema on the backend, while it also handles the ODM (Object-Document Mapping) or ORM (Object-Relational Mapping) mapping between data model and storage.

Next:

1. Create a New Loopback Application,
2. Create the Data Models from the Class Diagram,
3. Add Relations to the Models,
4. Define the Access Control List (ACL),
5. Test the Application,
6. Push Code to Github,
7. Add an Application Manifest for Deployment,
8. Add Continuous Integration to Cloud.

1. Create a New Loopback Appication

Open a command-line or Terminal and change to your development folder. Create a new Loopback project ‘myLoopback’ or ‘qavideos’ with the ‘lb’ command to run the Loopback cli. When the new project is successfully created, change the active folder to your new project folder, and test the new app.

$ cd dev/src/projects
$ lb
? What's the name of your application? qavideos
? Enter name of the directory to contain the project: qavideos
   create qavideos/
     info change the working directory to qavideos
? Which version of LoopBack would you like to use? 3.x (current)
? What kind of application do you have in mind? api-server (A LoopBack API server with local User auth)
Generating .yo-rc.json
I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.
create .editorconfig
create .eslintignore
create .eslintrc
create server/boot/root.js
create server/middleware.development.json
create server/middleware.json
create server/server.js
create README.md
create server/boot/authentication.js
create .gitignore
create client/README.md
Next steps:
Change directory to your app
  $ cd qavideos
Create a model in your app
  $ lb model
Run the app
  $ node .
$ cd qavideos
$ node .
Web server listening at: http://localhost:3000
Browse your REST API at http://localhost:3000/explorer

After the project is successfully created, you have a Node.js based application in the ‘myLoopback’ or ‘qavideos’ folder. Change to the project folder and start with the default command ‘ node . ’. Open the ‘explorer’ tool via http://localhost:3000/explorer.

To sign up as a new user, click to unfold the POST /Users endpoint. In the ‘Credentials’ parameter paste a new json object and click the ‘Try it out!’ button.

{
"email":"user1@email.com",
"password":"passw0rd",
"username":"user1"
}

The Response Body should return a new id.

{   
"username": "user1",   
"email": "user1@email.com",   
"id": 1 
}

Then to login click and unfold the POST /Users/login endpoint, in the ‘Credentials’ parameter paste the json object with the email and password properties, and click the ‘Try it out!’ button.

{
"email":"user1@email.com",
"password":"passw0rd"
}

The Response Body should return an id value that is the access token, which you use to authenticate other API requests.

{   
"id": "cwotd42A6VxOrWbFEN1EQigITKA1RnFHGWAKUz",   
"ttl": 1209600,   
"created": "2017-09-20T21:03:59.305Z",   
"userId": 1 
}

By default, not all endpoints are allowed for User extensions. To see what endpoints exactly are allowed browse to the ‘~/node_modules/loopback/common/models/user,json’ file and review the ‘acls’ property. I will talk about ACLs (Access Control Lists) later.

2. Create the Data Models from the Class Diagram

First, take a look at the class diagram for the QAVideos application.

The first thing to note about my class diagram, is that I am extending a User class. Loopback has a built-in User and Group model, so authentication ships with your new project out-of-the-box. But I want to add specific user properties to the User model like a user level or user type, and later I also want to allow 3rd party authentication tools like IBM App ID, Google Firebase Authentication, Twitter Sign-in, Amazon Cognito, Facebook Login, or Instagram Authentication. To use 3rd party authentication, I want to track these users by synchronizing their authentication type and information, and so that I can also access their social graphs if available. In the class diagram, this class extension is represented by a Generalization from Parent to Child, from User to UserExt class.

The second thing to note, is that the Video class is the Parent of the Question and Answer class. Alternatively, I could have just added a string property to Video that represents a qaType of either ‘question’ or ‘answer’.

The last thing to note, is that I have a Sticker class, which I use to represent later in the project all kind of feedback from the user: likes, dislikes, votes, smileys and other emojis. Remember that a comment is also user feedback, but in the QA model, I want to force users to screenrecord comments, so that a comment technically is nothing more than another Question or another Answer.

To create the model in Loopback, run the command ‘lb model’ and configure the classes in the class diagram. For now, I will select the in-memory database ‘db (memory)’ as the datasource, but later I will replace it by MongoDb. For independent classes, you have to select PersistedModel as the base class, but for a Child class of a Parent class, you choose the Parent as the base class.

Start with the UserExt model, and as base class scroll down to select the built-in User model.

UserExt.

lb model
? Enter the model name: UserExt
? Select the datasource to attach UserExt to: db (memory)
? Select model's base class User
? Expose UserExt via the REST API? Yes
? Custom plural form (used to build REST URL):
? Common model or server only? common

This will generate 2 files in the ‘~/common/models/’folder: ‘user-ext.js’ and ‘user-ext.json’, respectively the model function hook to overwrite default model events and the model configuration file. It also added a property for the new model in the ‘~/server/model-config.json’ file.

Edit the the ‘~/server/model-config.json’ file to hide the built-in User class, because we want to make the UserExt class public, and hide the User class.

"User": {
  "dataSource": "db",
  "public": false
}

Restart the application and make sure you now only see UserExt in the explorer.

Implement the other models: Sticker, Video, Question and Answer both extending Video.

Sticker.

$ lb model
? Enter the model name: Sticker
? Select the datasource to attach Sticker to: db (memory)
? Select model's base class PersistedModel
? Expose Sticker via the REST API? Yes
? Custom plural form (used to build REST URL):
? Common model or server only? common
Let's add some Sticker properties now.
Enter an empty property name when done.
? Property name: stickerType
invoke   loopback:property
? Property type: string
? Required? Yes
? Default value[leave blank for none]:
Let's add another Sticker property.
Enter an empty property name when done.
? Property name:

Video. Do not expose Video via the REST API, because we want to expose the extended models Question and Answer instead.

$ lb model
? Enter the model name: Video
? Select the datasource to attach Video to: db (memory)
? Select model's base class PersistedModel
? Expose Video via the REST API? No
? Custom plural form (used to build REST URL):
? Common model or server only? common
Let's add some Video properties now.
Enter an empty property name when done.
? Property name: title
invoke   loopback:property
? Property type: string
? Required? Yes
? Default value[leave blank for none]:
Let's add another Video property.
Enter an empty property name when done.
? Property name: url
invoke   loopback:property
? Property type: string
? Required? Yes
? Default value[leave blank for none]:
Let's add another Video property.
Enter an empty property name when done.
? Property name: description
invoke   loopback:property
? Property type: string
? Required? No
? Default value[leave blank for none]:
Let's add another Video property.
Enter an empty property name when done.
? Property name: datePublished
invoke   loopback:property
? Property type: string
? Required? No
? Default value[leave blank for none]:
Let's add another Video property.
Enter an empty property name when done.
? Property name:

Question.

$ lb model
? Enter the model name: Question
? Select the datasource to attach Question to: db (memory)
? Select model's base class Video
? Expose Question via the REST API? Yes
? Custom plural form (used to build REST URL):
? Common model or server only? common
Let's add some Question properties now.
Enter an empty property name when done.
? Property name:

Answer.

$ lb model
? Enter the model name: Answer
? Select the datasource to attach Answer to: db (memory)
? Select model's base class Video
? Expose Answer via the REST API? Yes
? Custom plural form (used to build REST URL):
? Common model or server only? common
Let's add some Answer properties now.
Enter an empty property name when done.
? Property name:

If you open the ‘~/server/model-config.json’ file, you should see the class diagram now reflected. Note that the parent models are hidden.

"User": {
  "dataSource": "db",
  "public": false
},
"UserExt": {
  "dataSource": "db",
  "public": true
},
"Sticker": {
  "dataSource": "db",
  "public": true
},
"Video": {
  "dataSource": "db",
  "public": false
},
"Question": {
  "dataSource": "db",
  "public": true
},
"Answer": {
  "dataSource": "db",
  "public": true
}

The ‘datasource’ property is referring to ‘db’. This ‘db’ is defined in the file ‘~/server/datasources.json’. The ‘db’ datasource is currently using a ‘memory’ connector. In a next tutorial I will replace this with a Loopback connector for MongoDB.

{
  "db": {
    "name": "db",
    "connector": "memory"
  }
}

3. Add Relations to the Models

We have the models defined, but what is still missing are the relations between the classes or models. If you look at the UML class diagram you see four associations:

• Question ‘belongs to’ UserExt,
• Answer ‘belongs to’ UserExt,
• Video ‘has many’ Stickers, and
• Question ‘has many’ Answers.

You may think of belongsTo as a relation that is the reversed of hasMany, but later in the tutorial we will see that in order for a UserExt or User to qualify to have access rights to a model, the model must have a belongsTo relation.

To create a relation in Loopback, run the command ‘lb relation’.

Question belongsToUserExt.

$ lb relation
? Select the model to create the relationship from: Question
? Relation type: belongs to
? Choose a model to create a relationship with: UserExt
? Enter the property name for the relation: belongsToUserExt
? Optionally enter a custom foreign key:
? Require a through model? No

Answers belongsToUserExt

$ lb relation
? Select the model to create the relationship from: Answer
? Relation type: belongs to
? Choose a model to create a relationship with: UserExt
? Enter the property name for the relation: belongsToUserExt
? Optionally enter a custom foreign key:
? Require a through model? No

Video hasManyStickers.

$ lb relation
? Select the model to create the relationship from: Video
? Relation type: has many
? Choose a model to create a relationship with: Sticker
? Enter the property name for the relation: hasManyStickers
? Optionally enter a custom foreign key:
? Require a through model? No

Question hasManyAnswers.

$ lb relation
? Select the model to create the relationship from: Question
? Relation type: has many
? Choose a model to create a relationship with: Answer
? Enter the property name for the relation: hasManyAnswers
? Optionally enter a custom foreign key:
? Require a through model? No

Open the model configuration file for the models Video, Question and Answer to see the changes made to the ‘relations’ property. If you open the configuration file for the Question model ‘~/common/models/question.json’ you see the following changes to the ‘relations’ property.

"relations": {
  "hasManyAnswers": {
    "type": "hasMany",
    "model": "Answer",
    "foreignKey": ""
  },
  "belongsToUserExt": {
    "type": "belongsTo",
    "model": "UserExt",
    "foreignKey": ""
  }
},

Restart the application with ‘node .’ and open the explorer again.

Try to recreate the new UserExt. Because we restarted the application, the memory database was reset and the UserExt instance has been cleared. Create a Question for the new UserExt, create an Answer for the Question, and try to query the Answers for a Question with the endpoint ‘GET /Questions/{id}/hasManyAnswers’.

curl -X GET --header 'Accept: application/json' 'http://localhost:3000/api/Questions/1/hasManyAnswers?access_token=<access_token>'

4. Define the Access Control List (ACL)

The class diagram is now implemented and you can access each model via a REST API endpoint. You can query for data using the REST API endpoints and the built-in query language in Loopback. Loopback 3.0 Queries are based on the Node.js API QueryString module. It includes a Fields filter and a Where filter, which gives you very powerful out of the box query functions similar to SQL WHERE clauses.

But, currently anyone can Create, Read, Update and Delete (CRUD) any of the data models’ data, because there are no restrictions or access controls defined. User A can delete data from User B, which is not desirable. To apply proper security, I will define Access Control Lists (ACL) in Loopback that will restrict rights to Create, Read, Update and Delete and control access to data for certain types of users.

Loopback maps CRUD to three different access types: READ, WRITE, and EXECUTE.

The READ access type corresponds to five different Loopback functions:

• exists,
• findById,
• find,
• findOne,
• count.

The WRITE access type corresponds to four different Loopback functions:

• create,
• updateAttributes,
• upsert,
• destroyById.

The EXECUTE access type corresponds to all other Loopback functions.

The access types are mapped to a User and their user rights via a Role. LoopBack provides the following built-in dynamic roles:

• $owner, owner of the object. To qualify a $owner, the target model needs to have a ‘belongsTo’ relation to the User model (or a model extends from User),
• $authenticated, authenticated user,
• $unauthenticated, unauthenticated user,
• $everyone, everyone.

You can also define custom static roles.

To define an ACL for a model, for each model run the command ‘lb acl’. A typical ACL for a model would consist of the following access types:

1. ‘Explicitly deny access’ to everyone or ‘All users’ ‘All’ access types (READ, WRITE, EXECUTE, and ‘All methods and properties’), often the starting point for each ACL,
2. Then, allow or ‘Explicitly grant access’ to everyone or ‘All users’ to ‘READ’ (‘All methods and properties’, see five Loopback functions under READ),
3. Allow or ‘Explicitly grant access’ to ‘Any authenticated user’ to ‘create’ (‘A single method’ under WRITE),
4. Allow or ‘Explicitly grant access’ to the owner to make changes or delete.

Question.

$ lb acl
? Select the model to apply the ACL entry to: Question
? Select the ACL scope: All methods and properties
? Select the access type: All (match all types)
? Select the role All users
? Select the permission to apply Explicitly deny access
$ lb acl
? Select the model to apply the ACL entry to: Question
? Select the ACL scope: All methods and properties
? Select the access type: Read
? Select the role All users
? Select the permission to apply Explicitly grant access
$ lb acl
? Select the model to apply the ACL entry to: Question
? Select the ACL scope: A single method
? Enter the method name create
? Select the role Any authenticated user
? Select the permission to apply Explicitly grant access
$ lb acl
? Select the model to apply the ACL entry to: Question
? Select the ACL scope: All methods and properties
? Select the access type: Write
? Select the role The user owning the object
? Select the permission to apply Explicitly grant access

Open the configuration file for Question ‘~/common/models/question.json’ and look at the ‘acls’ property.

"acls": [
{
  "accessType": "*",
  "principalType": "ROLE",
  "principalId": "$everyone",
  "permission": "DENY"
},
{
  "accessType": "READ",
  "principalType": "ROLE",
  "principalId": "$everyone",
  "permission": "ALLOW"
},
{
  "accessType": "EXECUTE",
  "principalType": "ROLE",
  "principalId": "$authenticated",
  "permission": "ALLOW",
  "property": "create"
},
{
  "accessType": "WRITE",
  "principalType": "ROLE",
  "principalId": "$owner",
  "permission": "ALLOW"
}
],

Do the same for Answer and Sticker. You can copy the ‘acls’ properties from Question and add it to ‘Answer.json’ and to ‘Sticker.json’.

5. Test the Application

As a new user, create a User1, and login.

{
"email":"user1@email.com",
"password":"passw0rd",
"username":"user1"
}

As User1, create a Question1.

{
  "title": "Is this Question One?",
  "url": "http://youtube.com/q/1",
  "description": "question 1",
  "datePublished": "2017-09-22",
  "belongsToUserExtId": 1
}

As User1, logout.

As a new user, create a User2, and login.

{
"email":"user2@email.com",
"password":"passw0rd",
"username":"user2"
}

As User2, create an Answer1 to the Question1 by User1

{
  "title": "Yes that Is Question One!",
  "url": "http://youtube.com/q/1/a/1",
  "description": "Answer 1",
  "datePublished": "2017-09-22",
  "belongsToUserExtId": 2,
  "questionId": 1
}

As User2, create a new Sticker to Question1.

{
  "stickerType": "Smiley",
  "questionId": 1
}

As User2, logout.

As unauthenticated user, try to create a new Answer to Question 1.

{
  "title": "Yessss Question One",
  "url": "http://youtube.com/q/1/a/2",
  "description": "Answer 2 to Question 1",
  "datePublished": "2017-09-22",
  "questionId": 1
}

Trying to create a new Answer as an unauthenticated user should return an HTTP status code of 401 Unauthorized.

6. Push Code to Github

Now our application is finished and I will add the project to a new code repository on Github.

Open a new command-line window or new Terminal, and change the present working directory to the project folder. Then initialize the project folder to setup git to track changes, add all files to the index, and commit changes.

$ cd ~/dev/src/projects/qavideos
$ git init
  Initialized empty Git repository in /Users/user/dev/src/projects/qavideos/.git/
$ git add .
$ git commit -m "first commit"

Then go to https://github.com, and login to your Github account. Create a new repository ‘myLoopback’.

Copy the Github URL, e.g. ‘https://github.com/remkohdev/myLoopback.git’. Then add our local Git repository ‘origin’ to the remote ‘master’ repository on Github, and push the indexed and committed changes.

$ git remote add origin https://github.com/remkohdev/myLoopback.git
$ git push -u origin master
Counting objects: 35, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (30/30), done.
Writing objects: 100% (35/35), 43.80 KiB | 0 bytes/s, done.
Total 35 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), done.
To https://github.com/remkohdev/myLoopback.git
* [new branch]      master -> master
Branch master set up to track remote branch master from origin.

Check that the code was pushed to the Github repository.

7. Add an Application Manifest for Deployment

There is 1 important thing missing in our setup. In a modern Agile setup, we want to setup our systems development life cycle (SDLC) in such a way that when we push code to our repository, an automated process of build, test and deployment is triggered. This process is called automated build, test and deployment or in other words Continuous Integration (CI).

I will setup the CI to build, test and deploy QAVideos on a Cloud, in this case, I use IBM Bluemix. IBM Bluemix uses an Infrastructure as a Service (IaaS) software called CloudFoundry. CloudFoundry lets you automate the deployment of an application by adding an Application Manifest named ‘manifest.yml’.

In the root directory of the project add a new file ‘manifest.yml’. Edit the file ‘manifest.yml’.

---
applications:
- name: <application-name>
  host: <application-name>
  memory: 1024M
  path: .
  instances: 1
  domain: mybluemix.net

Replace <application-name> by ‘myLoopback’. Note, that the value for the host property has to be a unique subdomain name on the domain (e.g. mybluemix.net). Add the manifest.yml file to the git index, commit changes, and push to the Github repository.

$ git add .
$ git commit -m "add cf manifest"
$ git push

8. Add Continuous Integration to Cloud

Our code is complete. All we need to do now, is to setup the Continuous Integration on the Cloud.

To deploy the application above to IBM Bluemix, you need a free IBM Bluemix account. If you are a student or work at a university you can create a university code to send you an academic extension code for IBM Bluemix at http://onthehub.com/ibm.

The quickest way is to create a pipeline on IBM Bluemix.

• Go to https://console.ng.bluemix.net/devops,
• Select ‘Pipelines’ in the left menu, or directly link to https://console.ng.bluemix.net/devops/pipelines/dashboard
• Click the ‘Create pipeline’ button, and select ‘Delivery Pipeline’,
• Optionally, you can change the name of the pipeline to ‘pipeline-myLoopback’,
• Scroll down to ‘Source Code’, and for ‘Git provider’, select ‘Github.com’,
• Under ‘Repository Type’, select ‘Link to an existing repository’,
• And under ‘My Repos’ select the Github repo with ‘myLoopback’’ application that includes the manifest.yml file,
• Click the ‘Create’ button,
• A new pipeline with a new toolchain is created, and you are taken to the toolchain page,
• In the toolchain page, in the ‘DELIVER’ window, click the ‘Delivery Pipeline’ icon,
• Try running the ‘BUILD’ stage.

The Build succeeds, but most likely your Deploy failed. If you look at the deploy logs, it says ‘Server error, status code: 400, error code: 210003, message: The host is taken: myLoopback’. Someone already used the host or subdomain ‘myloopback.mybluemix.net.’

Go back to the Delivery Pipeline page with the Build and Deploy stage. Now, change the ‘host’ property in the manifest.yml file to ‘<yourusername>-myLoopback’, and push changes to Github. Keep your other eye on the Delivery Pipeline page, because if all goes well, your Build is automatically triggered by pushing code to Github now.

Now your Deploy stage should succeed, and your application is now deployed. In the bottom of the Deploy stage, you see the URL for your application. Click on the URL to open your application, and append ‘/explorer’, to go to the explorer window of your API server. You can repeat the tests to create new users and data.

Now, every time you make a code change and push your change to your Github repository, your CI triggers the automated build and deployment. Your Agile environment and API server is complete!

Note, that what is missing here still are the automated tests, and we also did not yet connect Loopback to a permanent storage, but we are still using the in-memory database.