remko de knikker
Nov 3, 2018 · 11 min read

Objective: in this tutorial I will create a copy of the FDIC Bank Data Developer Portal using a Test Driven Development (TDD) architecture for an agile development team.

My version of the FDIC Bank Data Developer Portal is a Loopback based Node.js application generated from a Swagger, or Open API Specification, file with 2 endpoints to access FDIC Banks data.

When I submit new code to Github, the code is automatically tested using Postman and Newman using a Scripted Pipeline with Groovy in Jenkins to set up the Continuous Integration (CI).

  1. Requirements
  2. Create an API Application with Swagger
  3. Dockerize the Application
  4. Create Postman Tests
  5. Automate Testing with Newman
  6. Add Source Code to Github Repository
  7. Setup a Local Jenkins Server
  8. Add a Jenkins Pipeline to Automate Testing

1. Requirements

  • Docker
  • Node.js
  • Loopback 3
  • Postman
  • Newman
  • Github Account

2. Create an API Application with Swagger

The API Application accesses FDIC Institutions data (I used the current version of 2018–10–25). Below is a subset of one bank record with the most interesting data. The ‘{“docs”:[]}’ is a CouchDB bulk insert required format.

{
"docs": [
{
"STNAME": "Maryland",
"ACTIVE": 0,
"ADDRESS": "7720 York Road",
"ASSET": "1,446,884",
"CITY": "Towson",
"COUNTY": "Baltimore",
"FED_RSSD": 337322,
"NAME": "Citibank (Maryland), National Association",
"ZIP": 21204,
"WEBADDR": "www.jpmorganchase.com",
"OFFICES": 13,
"CITYHCR": ""
},
...
]
}

In my earlier Medium article ‘Java: Create a Spring Data JPA App with IBM Db2 and FDIC Institutions Data’, I created a Java Spring application that retrieves and accesses the full FDIC Institutions database stored in a DB2 instance. For the data fans, doing the data bulk insert of this CouchDB article and that DB2 article, will give you a quick insight into one advantage of a NoSQL database. In a real world scenario you could connect this API via Kafka or RabbitMQ to the Java Spring application.

Let’s get started.

  • Create a project directory ‘fdic-api’
  • Create the following sub directories: ‘api’, ‘data’, and ‘postman’
$ mkdir fdic-api
$ cd fdic-api
$ mkdir api
$ mkdir data
$ mkdir postman
$ mkdir couchdb-data
$ mkdir jenkins_home
$ lb fdic-banks-api
? What's the name of your application? fdic-banks-api
? Enter name of the directory to contain the project: src
create src/
info change the working directory to src
? Which version of LoopBack would you like to use? 3.x (Active Long Term Support)
? What kind of application do you have in mind? api-server (A LoopBack API server with local User auth)
  • [optional] rename ‘fdic-banks-api’ in ‘package.json’ and ‘package-lock.json’ to ‘fdic-api’. Because the project directory name was named fdic-api, I had to temporarily name the Loopback subdirectory something different, but I renamed it to ‘src’ after completion.
$ mv fdic-banks-api src
  • Add a .gitignore file to exclude certain files and directories from source code version management
$ vi .gitignore
.DS_Store
src/node_modules/
couchdb-data/

:wq
  • Generate the model from our ‘swagger.yaml’ file
$ cd src
$ lb swagger
? Enter the swagger spec url or file path: ../api/swagger.yaml
Loading ../api/swagger.yaml...
? Select models to be generated: (Press <space> to select, <a> to toggle all, <i> to inverse selection)Bank, Banks
? Select the datasource to attach models to: db (memory)
Creating model definition for Bank...
Creating model definition for Banks...
Model definition created/updated for Banks.
Model definition created/updated for Bank.
Creating model config for Bank...
Creating model config for Banks...
Model config created for Bank.
Model config created for Banks.
Generating /Users/remkohdev/dev/src/fdic-api/src/common/models/banks.js
Models are successfully generated from swagger spec.
  • Run the application and browse the APIs, you will see the Banks API and a built-in User API.
$ node .
Web server listening at: http://localhost:3000
Browse your REST API at http://localhost:3000/explorer
  • Setup CouchDB
$ cd ..
$ docker run -d --restart always --hostname localhost --name couchdb -p 5984:5984 -e COUCHDB_USER=couchdbadmin -e COUCHDB_PASSWORD=couchdbadmin -v "$(pwd)"/couchdb-data:/opt/couchdb/data couchdb
Unable to find image 'couchdb:latest' locally
latest: Pulling from library/couchdb
  • Create a database ‘fdic_institutions2’
$ curl -X PUT http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2
$ curl -X POST -H "Content-Type: application/json" "http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_bulk_docs" -d '@data/INSTITUTIONS2.json'
  • Create necessary views in CouchDB
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"name-view": {
"map": "function(doc){ if(doc.NAME){ emit(doc.NAME, doc); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/name
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"asset-view": {
"map": "function(doc){ if(doc.ASSET){ emit(parseInt(doc.ASSET.replace(\/,\/g, \"\")), doc); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/asset
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"name-asset-view": {
"map": "function(doc){ if(doc.NAME && doc.ASSET){ emit(doc.NAME, doc.ASSET); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/name-asset
$ curl -X PUT -H "Content-Type: application/json" -d '{
"views": {
"fed-rssd-view": {
"map": "function(doc){ if(doc.FED_RSSD){ emit(doc.FED_RSSD); }}"
}
},
"language": "javascript"
}' http://couchdbadmin:couchdbadmin@localhost:5984/fdic_institutions2/_design/fed-rssd
  • Configure couchdb connector in ‘src/server/datasources.json’
{
"db": {
"name": "db",
"connector": "memory"
},
"couchdb": {
"name": "couchdb",
"connector": "couchdb2",
"url": "http://couchdbadmin:couchdbadmin@localhost:5984",
"database": "fdic_institutions2"
}
}
  • Install connector, juggler, util, lodash packages. The first three are required by the couchdb connector. Lodash is a utility function I use for request parameter validation.
$ cd src
$ npm install --save util
$ npm install --save loopback-datasource-juggler
$ npm install --save loopback-connector-couchdb2
$ npm install --save lodash
  • Edit the ‘src/common/models/banks.js’ in order to be able to query FDIC Institutions data
$ npm start
> fdic-api@1.0.0 start /Users/remkohdev@us.ibm.com/dev/src/medium/fdic-api/src
> node .
Web server listening at: http://localhost:3000
Browse your REST API at http://localhost:3000/explorer

3. Dockerize the Application

  • Make sure you stop the node app from the previous step, cause it will block the 3000 port,
  • Create a new file ‘Dockerfile’,
$ cd ..
$ vi Dockerfile
FROM node:latestWORKDIR /usr/src/appCOPY ./src/package*.json ./# install dependencies
RUN npm install
# automatic security fixes
RUN npm audit fix
# manual security fixes
# Bundle app source
COPY ./src .
EXPOSE 3000CMD [ "npm", "start" ]:wq
  • Note, that we configured the data source for couchdb to ‘localhost’. Inside a container that would resolve to it self, so you need to get the IP address of your host machine and replace the data source host by that IP address,
$ ipconfig getifaddr en0
192.168.1.5
  • Edit the ‘src/server/datasources.json’ file,
"couchdb": {
"name": "couchdb",
"connector": "couchdb2",
"url": "http://couchdbadmin:couchdbadmin@192.168.1.5:5984",
"database": "fdic_institutions2"
}
  • Create a bash script file ‘docker-run.sh’,
$ vi docker-run.sh#!/bin/bash -xvecho '=====>docker stop'
docker stop fdic-api
docker rm fdic-api
echo '=====>docker build'
docker build --no-cache -t fdic-api .
echo '=====>docker run'
docker run -d --restart always --name fdic-api -p 3000:3000 -e "NODE_ENV=local" fdic-api
:wq
  • Run the ‘docker-run.sh’ script,
$ sh docker-run.sh
a4d6a2cd7685d0c294f874d0f16d1740bbdffdd3c64d7e893c7099885589795d
$ $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e72a52caf3c9 fdic-api "npm start" 36 seconds ago Up 35 seconds 0.0.0.0:3000->3000/tcp fdic-api
5821ad6884b2 couchdb "tini -- /docker-ent…" 41 minutes ago Up 41 minutes 4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp couchdb

4. Create Postman Tests

Postman is a great way to create E2E tests for APIs. We have created 2 API endpoints that we need to test:

  • /banks/{fed rssd}
  • /banks?asset={minimum asset}&name={bank name}

Download the collection and environment your project directory ‘postman’ from

Or access the published collection via the documenter.getpostman.com site

  • https://documenter.getpostman.com/view/305433/RzZ3N3Z1
  • To run the tests manually in Postman, open Postman,
  • Click Import > Import File > Choose Files
  • Select FDIC-API_LOCALHOST.postman_environment.json
  • Click the Manage Environments Settings icon
  • In Manage Environments > Import > Choose Files
  • Select FDIC-API_LOCALHOST.postman_environment.json
  • Set the Environment to ‘FDIC-API.LOCALHOST’
  • Select the FDIC_API > NORMAL > GET /banks/{id} test
  • Select the Tests tab, here is where some of the Postman tests are scripted, e.g. “Check institution.NAME available”

5. Automate Testing with Newman

  • Run the postman collection from the commandline with Newman
$ newman run 'postman/FDIC-API.postman_collection.json' -e 'postman/FDIC-API_LOCALHOST.postman_environment.json' --timeout-request 5000
newman
FDIC_API❏ NORMAL
↳ /banks/{id}
GET http://192.168.1.5:3000/api/Banks/3150447 [200 OK, 2.79KB, 77ms]
✓ Check Status code is 200
✓ Check institution.NAME available
✓ Check institution.NAME available
# failure detail1. Error ESOCKETTIMEDOUT
at request
inside "EXCEPTIONS / /banks?assets= - assets not a number"
  • Note that there are only 3 assertions that we really are testing for, all defined in the GET /banks/{id} request, the other requests have no assertions and hence no true tests assigned, but even so, the GET /banks?assets= request times out given the 5 seconds (5000 ms) timeout I have set. The 3 true tests all passed successfully.

6. Add Source Code to Github Repository

  • To create a new Github repository for your project, go to https://github.com/<username>?tab=repositories, click the ‘New’ button
  • For repository name enter ‘fdic-api’,
  • For description enter ‘api for fdic institutions data’,
  • Make the repository public to share,
  • Uncheck the README.md (add your own),
  • Do not add a .gitignore,
  • Do not add a license,
  • Click ‘Create repository’
  • You should see instructions how to add your local repository to Github
  • Copy the URL for the new remote repository on Github, e.g. git@github.com:remkohdev/fdic-api.git or https://github.com/remkohdev/fdic-api.git
  • Add your local project to Github
$ git init
Initialized empty Git repository in /Users/me/dev/src/fdic-api/.git/
$ git add .
$ git commit -m "first commit"
$ git remote add origin git@github.com:<username>/fdic-api.git
$ git push -u origin master
Counting objects: 43, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (37/37), done.
Writing objects: 100% (43/43), 2.19 MiB | 1.89 MiB/s, done.
Total 43 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'master' on GitHub by visiting:
remote: https://github.com/remkohdev/fdic-api/pull/new/master
remote:
To github.com:remkohdev/fdic-api.git
* [new branch] master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
  • If your Github repository is private, then you need to create a personal access token on Github, as described https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
  • Go to Profile > Settings > Developer Settings > Personal Access Tokens
  • Generate a new token with description ‘github-personal-access-token’,
  • Check the ‘repo’ and ‘user:email’ scope
  • Click ‘Generate token’
  • Save the personal access token, you need it in Jenkins
  • If your Github repository is public, then you do not need a personal access token,

7. Setup a Local Jenkins Server

You can install the Jenkins server in several ways. I am not a Jenkins configuration expert, so if you run into problems, try one of the other methods or google your way out, unfortunately I won’t be much help if you do.

  1. docker run
  2. homebrew install jenkins
  3. java -jar

Docker Run

$ docker run -d --restart always --name jenkins -p 8080:8080 -p 50000:50000 -v ${PWD}/jenkins_home:/var/jenkins_home jenkins/jenkins:lts
Unable to find image 'jenkins/jenkins:lts' locally
lts: Pulling from jenkins/jenkins
  • To configure the newly running Jenkins server, we need to retrieve the initial admin password from within the container,
$ docker exec -ti jenkins sh -c /bin/bash
jenkins@b5730aba6465:/$ cat /var/jenkins_home/secrets/initialAdminPassword
46065766e4e299a9820a9a18c16015bb
jenkins@b5730aba6465:/$ exit
exit
$

Homebrew Install

Install the Long Term Support version of Jenkins with Homebrew:

$ brew install jenkins-lts

Java -Jar

$ java -jar jenkins.war --httpPort=8080

Configure Jenkins

  • Open a new browser tab or window and go to http://localhost:8080/ use the initialAdminPassword as ‘Administrator password’ to ‘Unlock Jenkins’.
  • On the ‘Customize Jenkins’ page, install suggested plugins.
  • Create a first admin user. Try to avoid using default values that will become security vulnerabilities later.
username: jenkinsadmin
password: passw0rd
  • Configure your instance at: http://localhost:8080/
  • Save and continue, Save and finish, Start using Jenkins
  • Or continue as admin,
  • The Git plugin and the Github plugin are installed if you installed suggested plugins.
  • Manage Jenkins > Manage Plugins > Available > search ‘Blue Ocean’
  • Install the ‘Blue Ocean, Aggregator’ plugin without restart, this will install all Blue Ocean modules.
  • Manage Plugins > Available > NodeJS
  • Install the ‘NodeJS’ plugin without restart.
  • Restart Jenkins
  • Go to Jenkins > Manage Jenkins > Global Tool Configuration > NodeJS > Add NodeJS, and select the latest 10.x version, which according to the LTS (Long Term Support) Schedule is active on 2018–10–30. Enter a NodeJS Name, e.g. ‘NodeJS 10.13.0', ensure that ‘Install automatically’ is selected. Note: the NodeJS Name ‘NodeJS 10.13.0’ is later referenced in the Jenkinsfile ‘withEnv’ syntax.
  • For ‘Global npm packages to install’, enter ‘newman’
  • Save the new Global Tool Configuration.

8. Add a Jenkins Pipeline to Automate Testing

  • Verify that our fdic-api, couchdb, and jenkins containers are running
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec8a7eda5ad6 fdic-api "npm start" 3 hours ago Up 3 hours 0.0.0.0:3000->3000/tcp fdic-api
d471529cad10 jenkins/jenkins:lts "/sbin/tini -- /usr/…" 3 days ago Up 3 days 0.0.0.0:50000->50000/tcp, 0.0.0.0:8080->8080/tcp jenkins
f5351a5bbfd5 couchdb "tini -- /docker-ent…" 2 weeks ago Up 2 weeks 4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp couchdb
  • Open Jenkins at http://localhost:8080
  • Click ‘New Item’ to get started
  • Enter a name: ‘fdic-api’,
  • Select ‘Pipeline’
  • Click ‘OK’
  • In General, check the ‘Github project’ box, in ‘Project url’ enter your Github repository URL, e.g. https://github.com/remkohdev/fdic-api.git
  • In the ‘Build Triggers’ section, check the ‘Poll SCM’ box, and in the ‘Schedule’ textarea type ‘H/5 * * * *’ to poll for changes every 5 minutes or ‘@hourly’ to poll for changes every hour,
  • In the Pipeline section, enter the following Pipeline script

Or use a Jenkinsfile from the Github repository,

  • In the ‘Pipeline’ section, for ‘Definition’ select ‘Pipeline script from SCM’, as SCM select ‘Git’,
  • In Repositories, for Repository URL enter your Github Repository URL, e.g. ‘https://github.com/remkohdev/fdic-api.git
  • If your Github repository is private, add a credential,
  • In ‘Credentials’, select ‘Add’, select ‘Jenkins’
  • Paste the ‘personal access token’ into the ‘Secret’ textbox
  • Paste ‘github-personal-access-token’ into the ID textbox and the ‘Description’ textbox,
  • If your Github repository is public do not add the credential,
  • Click Add
  • In ‘Branches to build’, enter ‘*/master’ if you use the master branch,
  • In Script Path make sure the relative path to the Jenkinsfile is entered, ‘Jenkinsfile’,
  • Save
  • Note: I had to use the withEnv syntax to resolve the NodeJS plugin not automatically being installed in the Jenkins runtime and prevent the script to fail with a ‘node unknown’ exception.
  • Click Build Now
  • To view the build history in Blue Ocean, click the ‘Open Blue Ocean’ menu item.
  • Wait to see if the automated test fails or succeeds. In my case, one of the requests times out.
/banks?assets= - assets not a number
GET http://192.168.1.5:3000/api/Banks/getBanks?assets=abc,def,ghi [errored]
ESOCKETTIMEDOUT
  • By default, newman reports within the cli
┌─────────────────────────┬──────────┬──────────┐
│ │ executed │ failed │
├─────────────────────────┼──────────┼──────────┤
│ iterations │ 1 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ requests │ 8 │ 1 │
├─────────────────────────┼──────────┼──────────┤
│ test-scripts │ 8 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ prerequest-scripts │ 8 │ 0 │
├─────────────────────────┼──────────┼──────────┤
│ assertions │ 3 │ 0 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 5.8s │
├───────────────────────────────────────────────┤
│ total data received: 71.82KB (approx) │
├───────────────────────────────────────────────┤
│ average response time: 44ms │
└───────────────────────────────────────────────┘

Done!

If you want to trigger a new build in Jenkins every time you submit code to Github, in Github for organization or repository, in Settings > Hooks, you need to create a webhook for individual events like ‘Pull requests’ and ‘Pushes’, set the ‘Payload URL’ to your Jenkins server (e.g. using Ngrok for localhost access), Content-Type to ‘application/x-www-form-urlencoded’, and check the ‘Active’ box.

NYC⚡️DEV

NYC Developer Community

remko de knikker

Written by

Developer Advocate @IBMDeveloper @IBMCloud — stateless tech humanist, serpent in the shepherds mouth — Dutch NY-er

NYC⚡️DEV

NYC Developer Community

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade