Agile and Test Driven Development (TDD) with Swagger, Docker, Github, Postman, Newman and Jenkins for a Loopback, Node.js and CouchDB App

remko de knikker
Nov 3, 2018 · 11 min read

Objective: in this tutorial I will create a copy of the FDIC Bank Data Developer Portal using a Test Driven Development (TDD) architecture for an agile development team.

My version of the FDIC Bank Data Developer Portal is a Loopback based Node.js application generated from a Swagger, or Open API Specification, file with 2 endpoints to access FDIC Banks data.

When I submit new code to Github, the code is automatically tested using Postman and Newman using a Scripted Pipeline with Groovy in Jenkins to set up the Continuous Integration (CI).

1. Requirements

  • Docker
  • Node.js
  • Loopback 3
  • Postman
  • Newman
  • Github Account

2. Create an API Application with Swagger

The API Application accesses FDIC Institutions data (I used the current version of 2018–10–25). Below is a subset of one bank record with the most interesting data. The ‘{“docs”:[]}’ is a CouchDB bulk insert required format.

{
"docs": [
{
"STNAME": "Maryland",
"ACTIVE": 0,
"ADDRESS": "7720 York Road",
"ASSET": "1,446,884",
"CITY": "Towson",
"COUNTY": "Baltimore",
"FED_RSSD": 337322,
"NAME": "Citibank (Maryland), National Association",
"ZIP": 21204,
"WEBADDR": "www.jpmorganchase.com",
"OFFICES": 13,
"CITYHCR": ""
},
...
]
}

In my earlier Medium article ‘Java: Create a Spring Data JPA App with IBM Db2 and FDIC Institutions Data’, I created a Java Spring application that retrieves and accesses the full FDIC Institutions database stored in a DB2 instance. For the data fans, doing the data bulk insert of this CouchDB article and that DB2 article, will give you a quick insight into one advantage of a NoSQL database. In a real world scenario you could connect this API via Kafka or RabbitMQ to the Java Spring application.

Let’s get started.

  • Create a project directory ‘fdic-api’
  • Create the following sub directories: ‘api’, ‘data’, and ‘postman’
$ mkdir fdic-api
$ cd fdic-api
$ mkdir api
$ mkdir data
$ mkdir postman
$ mkdir couchdb-data
$ mkdir jenkins_home
$ lb fdic-banks-api
? What's the name of your application? fdic-banks-api
? Enter name of the directory to contain the project: src
create src/
info change the working directory to src
? Which version of LoopBack would you like to use? 3.x (Active Long Term Support)
? What kind of application do you have in mind? api-server (A LoopBack API server with local User auth)
  • [optional] rename ‘fdic-banks-api’ in ‘package.json’ and ‘package-lock.json’ to ‘fdic-api’. Because the project directory name was named fdic-api, I had to temporarily name the Loopback subdirectory something different, but I renamed it to ‘src’ after completion.
$ mv fdic-banks-api src
  • Add a .gitignore file to exclude certain files and directories from source code version management
$ vi .gitignore
.DS_Store
src/node_modules/
couchdb-data/

:wq
  • Generate the model from our ‘swagger.yaml’ file
$ cd src
$ lb swagger
? Enter the swagger spec url or file path: ../api/swagger.yaml
Loading ../api/swagger.yaml...
? Select models to be generated: (Press <space> to select, <a> to toggle all, <i> to inverse selection)Bank, Banks
? Select the datasource to attach models to: db (memory)
Creating model definition for Bank...
Creating model definition for Banks...
Model definition created/updated for Banks.
Model definition created/updated for Bank.
Creating model config for Bank...
Creating model config for Banks...
Model config created for Bank.
Model config created for Banks.
Generating /Users/remkohdev/dev/src/fdic-api/src/common/models/banks.js
Models are successfully generated from swagger spec.
  • Run the application and browse the APIs, you will see the Banks API and a built-in User API.
$ node .
Web server listening at: http://localhost:3000
Browse your REST API at http://localhost:3000/explorer
  • Setup CouchDB
$ cd ..
$ docker run -d --restart always --hostname localhost --name couchdb -p 5984:5984 -e COUCHDB_USER=couchdbadmin -e COUCHDB_PASSWORD=couchdbadmin -v "$(pwd)"/couchdb-data:/opt/couchdb/data couchdb
Unable to find image 'couchdb:latest' locally
latest: Pulling from library/couchdb
  • Create a database ‘fdic_institutions2’
$ curl -X PUT http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2
$ curl -X POST -H "Content-Type: application/json" "http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_bulk_docs" -d '@data/INSTITUTIONS2.json'
  • Create necessary views in CouchDB
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"name-view": {
"map": "function(doc){ if(doc.NAME){ emit(doc.NAME, doc); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/name
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"asset-view": {
"map": "function(doc){ if(doc.ASSET){ emit(parseInt(doc.ASSET.replace(\/,\/g, \"\")), doc); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/asset
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"name-asset-view": {
"map": "function(doc){ if(doc.NAME && doc.ASSET){ emit(doc.NAME, doc.ASSET); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/name-asset
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"fed-rssd-view": {
"map": "function(doc){ if(doc.FED_RSSD){ emit(doc.FED_RSSD); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/fed-rssd
  • Configure couchdb connector in ‘src/server/datasources.json’
{
"db": {
"name": "db",
"connector": "memory"
},
"couchdb": {
"name": "couchdb",
"connector": "couchdb2",
"url": "http://couchdbadmin:couchdbadmin@localhost:5984",
"database": "fdic_institutions2"
}
}
  • Install connector, juggler, util, lodash packages. The first three are required by the couchdb connector. Lodash is a utility function I use for request parameter validation.
$ cd src
$ npm install --save util
$ npm install --save loopback-datasource-juggler
$ npm install --save loopback-connector-couchdb2
$ npm install --save lodash
  • Edit the ‘src/common/models/banks.js’ in order to be able to query FDIC Institutions data
$ npm start
> fdic-api@1.0.0 start /Users/remkohdev@us.ibm.com/dev/src/medium/fdic-api/src
> node .
Web server listening at: http://localhost:3000
Browse your REST API at http://localhost:3000/explorer

3. Dockerize the Application

  • Make sure you stop the node app from the previous step, cause it will block the 3000 port,
  • Create a new file ‘Dockerfile’,
$ cd ..
$ vi Dockerfile
FROM node:latestWORKDIR /usr/src/appCOPY ./src/package*.json ./# install dependencies
RUN npm install
# automatic security fixes
RUN npm audit fix
# manual security fixes
# Bundle app source
COPY ./src .
EXPOSE 3000CMD [ "npm", "start" ]:wq
  • Note, that we configured the data source for couchdb to ‘localhost’. Inside a container that would resolve to it self, so you need to get the IP address of your host machine and replace the data source host by that IP address,
$ ipconfig getifaddr en0
192.168.1.5
  • Edit the ‘src/server/datasources.json’ file,
"couchdb": {
"name": "couchdb",
"connector": "couchdb2",
"url": "http://couchdbadmin:couchdbadmin@192.168.1.5:5984",
"database": "fdic_institutions2"
}
  • Create a bash script file ‘docker-run.sh’,
$ vi docker-run.sh#!/bin/bash -xvecho '=====>docker stop'
docker stop fdic-api
docker rm fdic-api
echo '=====>docker build'
docker build --no-cache -t fdic-api .
echo '=====>docker run'
docker run -d --restart always --name fdic-api -p 3000:3000 -e "NODE_ENV=local" fdic-api
:wq
  • Run the ‘docker-run.sh’ script,
$ sh docker-run.sh
a4d6a2cd7685d0c294f874d0f16d1740bbdffdd3c64d7e893c7099885589795d
$ $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e72a52caf3c9 fdic-api "npm start" 36 seconds ago Up 35 seconds 0.0.0.0:3000->3000/tcp fdic-api
5821ad6884b2 couchdb "tini -- /docker-ent…" 41 minutes ago Up 41 minutes 4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp couchdb

4. Create Postman Tests

Postman is a great way to create E2E tests for APIs. We have created 2 API endpoints that we need to test:

  • /banks/{fed rssd}
  • /banks?asset={minimum asset}&name={bank name}

Download the collection and environment your project directory ‘postman’ from

Or access the published collection via the documenter.getpostman.com site

  • https://documenter.getpostman.com/view/305433/RzZ3N3Z1
  • To run the tests manually in Postman, open Postman,
  • Click Import > Import File > Choose Files
  • Select FDIC-API_LOCALHOST.postman_environment.json
  • Click the Manage Environments Settings icon
  • In Manage Environments > Import > Choose Files
  • Select FDIC-API_LOCALHOST.postman_environment.json
  • Set the Environment to ‘FDIC-API.LOCALHOST’
  • Select the FDIC_API > NORMAL > GET /banks/{id} test
  • Select the Tests tab, here is where some of the Postman tests are scripted, e.g. “Check institution.NAME available”

5. Automate Testing with Newman

  • Run the postman collection from the commandline with Newman
$ newman run 'postman/FDIC-API.postman_collection.json' -e 'postman/FDIC-API_LOCALHOST.postman_environment.json' --timeout-request 5000
newman
FDIC_API❏ NORMAL
↳ /banks/{id}
GET http://192.168.1.5:3000/api/Banks/3150447 [200 OK, 2.79KB, 77ms]
✓ Check Status code is 200
✓ Check institution.NAME available
✓ Check institution.NAME available
# failure detail1. Error ESOCKETTIMEDOUT
at request
inside "EXCEPTIONS / /banks?assets= - assets not a number"
  • Note that there are only 3 assertions that we really are testing for, all defined in the GET /banks/{id} request, the other requests have no assertions and hence no true tests assigned, but even so, the GET /banks?assets= request times out given the 5 seconds (5000 ms) timeout I have set. The 3 true tests all passed successfully.

6. Add Source Code to Github Repository

  • To create a new Github repository for your project, go to https://github.com/<username>?tab=repositories, click the ‘New’ button
  • For repository name enter ‘fdic-api’,
  • For description enter ‘api for fdic institutions data’,
  • Make the repository public to share,
  • Uncheck the README.md (add your own),
  • Do not add a .gitignore,
  • Do not add a license,
  • Click ‘Create repository’
  • You should see instructions how to add your local repository to Github
  • Copy the URL for the new remote repository on Github, e.g. git@github.com:remkohdev/fdic-api.git or https://github.com/remkohdev/fdic-api.git
  • Add your local project to Github
$ git init
Initialized empty Git repository in /Users/me/dev/src/fdic-api/.git/
$ git add .
$ git commit -m "first commit"
$ git remote add origin git@github.com:<username>/fdic-api.git
$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (37/37), done.
Writing objects: 100% (43/43), 2.19 MiB | 1.89 MiB/s, done.
Total 43 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'master' on GitHub by visiting:
remote: https://github.com/remkohdev/fdic-api/pull/new/master
remote:
To github.com:remkohdev/fdic-api.git
* [new branch] master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
  • If your Github repository is private, then you need to create a personal access token on Github, as described https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
  • Go to Profile > Settings > Developer Settings > Personal Access Tokens
  • Generate a new token with description ‘github-personal-access-token’,
  • Check the ‘repo’ and ‘user:email’ scope
  • Click ‘Generate token’
  • Save the personal access token, you need it in Jenkins
  • If your Github repository is public, then you do not need a personal access token,

7. Setup a Local Jenkins Server

You can install the Jenkins server in several ways. I am not a Jenkins configuration expert, so if you run into problems, try one of the other methods or google your way out, unfortunately I won’t be much help if you do.

Docker Run

$ docker run -d --restart always --name jenkins -p 8080:8080 -p 50000:50000 -v ${PWD}/jenkins_home:/var/jenkins_home jenkins/jenkins:lts
Unable to find image 'jenkins/jenkins:lts' locally
lts: Pulling from jenkins/jenkins
  • To configure the newly running Jenkins server, we need to retrieve the initial admin password from within the container,
$ docker exec -ti jenkins sh -c /bin/bash
jenkins@b5730aba6465:/$ cat /var/jenkins_home/secrets/initialAdminPassword
46065766e4e299a9820a9a18c16015bb
jenkins@b5730aba6465:/$ exit
exit
$

Homebrew Install

Install the Long Term Support version of Jenkins with Homebrew:

$ brew install jenkins-lts

Java -Jar

$ java -jar jenkins.war --httpPort=8080

Configure Jenkins

  • Open a new browser tab or window and go to http://localhost:8080/ use the initialAdminPassword as ‘Administrator password’ to ‘Unlock Jenkins’.
  • On the ‘Customize Jenkins’ page, install suggested plugins.
  • Create a first admin user. Try to avoid using default values that will become security vulnerabilities later.
username: jenkinsadmin
password: passw0rd
  • Configure your instance at: http://localhost:8080/
  • Save and continue, Save and finish, Start using Jenkins
  • Or continue as admin,
  • The Git plugin and the Github plugin are installed if you installed suggested plugins.
  • Manage Jenkins > Manage Plugins > Available > search ‘Blue Ocean’
  • Install the ‘Blue Ocean, Aggregator’ plugin without restart, this will install all Blue Ocean modules.
  • Manage Plugins > Available > NodeJS
  • Install the ‘NodeJS’ plugin without restart.
  • Restart Jenkins
  • Go to Jenkins > Manage Jenkins > Global Tool Configuration > NodeJS > Add NodeJS, and select the latest 10.x version, which according to the LTS (Long Term Support) Schedule is active on 2018–10–30. Enter a NodeJS Name, e.g. ‘NodeJS 10.13.0', ensure that ‘Install automatically’ is selected. Note: the NodeJS Name ‘NodeJS 10.13.0’ is later referenced in the Jenkinsfile ‘withEnv’ syntax.
  • For ‘Global npm packages to install’, enter ‘newman’
  • Save the new Global Tool Configuration.

8. Add a Jenkins Pipeline to Automate Testing

  • Verify that our fdic-api, couchdb, and jenkins containers are running
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec8a7eda5ad6 fdic-api "npm start" 3 hours ago Up 3 hours 0.0.0.0:3000->3000/tcp fdic-api
d471529cad10 jenkins/jenkins:lts "/sbin/tini -- /usr/…" 3 days ago Up 3 days 0.0.0.0:50000->50000/tcp, 0.0.0.0:8080->8080/tcp jenkins
f5351a5bbfd5 couchdb "tini -- /docker-ent…" 2 weeks ago Up 2 weeks 4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp couchdb
  • Open Jenkins at http://localhost:8080
  • Click ‘New Item’ to get started
  • Enter a name: ‘fdic-api’,
  • Select ‘Pipeline’
  • Click ‘OK’
  • In General, check the ‘Github project’ box, in ‘Project url’ enter your Github repository URL, e.g. https://github.com/remkohdev/fdic-api.git
  • In the ‘Build Triggers’ section, check the ‘Poll SCM’ box, and in the ‘Schedule’ textarea type ‘H/5 * * * *’ to poll for changes every 5 minutes or ‘@hourly’ to poll for changes every hour,
  • In the Pipeline section, enter the following Pipeline script

Or use a Jenkinsfile from the Github repository,

  • In the ‘Pipeline’ section, for ‘Definition’ select ‘Pipeline script from SCM’, as SCM select ‘Git’,
  • In Repositories, for Repository URL enter your Github Repository URL, e.g. ‘https://github.com/remkohdev/fdic-api.git
  • If your Github repository is private, add a credential,
  • In ‘Credentials’, select ‘Add’, select ‘Jenkins’
  • Paste the ‘personal access token’ into the ‘Secret’ textbox
  • Paste ‘github-personal-access-token’ into the ID textbox and the ‘Description’ textbox,
  • If your Github repository is public do not add the credential,
  • Click Add
  • In ‘Branches to build’, enter ‘*/master’ if you use the master branch,
  • In Script Path make sure the relative path to the Jenkinsfile is entered, ‘Jenkinsfile’,
  • Save
  • Note: I had to use the withEnv syntax to resolve the NodeJS plugin not automatically being installed in the Jenkins runtime and prevent the script to fail with a ‘node unknown’ exception.
  • Click Build Now
  • To view the build history in Blue Ocean, click the ‘Open Blue Ocean’ menu item.
  • Wait to see if the automated test fails or succeeds. In my case, one of the requests times out.
/banks?assets= - assets not a number
GET http://192.168.1.5:3000/api/Banks/getBanks?assets=abc,def,ghi [errored]
ESOCKETTIMEDOUT
  • By default, newman reports within the cli
┌─────────────────────────┬──────────┬──────────┐
│ │ executed │ failed │
├─────────────────────────┼──────────┼──────────┤
│ iterations │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ requests │ 8 │ 1 │
├─────────────────────────┼──────────┼──────────┤
│ test-scripts │ 8 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ prerequest-scripts │ 8 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ assertions │ 3 │ 0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 5.8s │
├───────────────────────────────────────────────┤
│ total data received: 71.82KB (approx) │
├───────────────────────────────────────────────┤
│ average response time: 44ms │
└───────────────────────────────────────────────┘

Done!

If you want to trigger a new build in Jenkins every time you submit code to Github, in Github for organization or repository, in Settings > Hooks, you need to create a webhook for individual events like ‘Pull requests’ and ‘Pushes’, set the ‘Payload URL’ to your Jenkins server (e.g. using Ngrok for localhost access), Content-Type to ‘application/x-www-form-urlencoded’, and check the ‘Active’ box.

NYC⚡️DEV

NYC Developer Community

remko de knikker

Written by

Developer Advocate @IBMDeveloper @IBMCloud — stateless tech humanist, serpent in the shepherds mouth — Dutch NY-er

NYC⚡️DEV

NYC Developer Community