Deploying your own private Node package manager to Bluemix

JavaScript has established a strong foothold in the enterprise development market. One of the reasons for this significant growth is the explosive expansion of the language to grow from being solely in the browser to becoming a mainstay in server-side runtimes with Node.js. One of the key drivers of programming language adoption is easy access to rich libraries that are developed by the programming community. Node.js is well known for their dependency management system called NPM that allows developers to share their code assets to the broader community. With its simplified command line interface, npm allows you to quickly install your favorite packages to incorporate foundational libraries such as express for building our RESTful applications and passport for building out secured applications backed by OAUTH.

Authors

Todd Kaplinger is a Senior Technical Staff Member and IBM Master Inventor in IBM’s Cloud Group. He is the Mobile Cloud Platform Architect focusing on delivering Mobile Cloud Services and has been one of IBM leading thought leaders and architects in mobile for the past 4 years. Todd is an expert in mobile application development in both Android and iOS as well as a strong history in web-based technologies such as Node and Java. In the past 14 years, Todd has been the lead architect on many IBM incubation projects and has participated in various Java Standards Groups.

Alex Weidner is a Staff Software Engineer that has been working in IBM Cloud since 2014, starting as a co-op with jStart and transitioning into a full time role with under the Mobile First organization, working on services such as Presence Insights. Alex bridges open source communities, practices, and software with internal concerns and requirements, continually pushing for adoption of modern practices internally and automating devops and development tasks.

Why

Despite this great model for sharing your public code assets with the broader community, there are always going to be times when your application leverages libraries that are specific to your solution and are not intended to be shared broadly. Examples include proprietary intellectual property that enterprises do not want exposed outside their company, governed updates so that system changes can be change controlled, speeding up the development and delivery pipeline, and common versions that can be internally shared and developed. In these instances, there are multiple ways to integrate these libraries with your application spanning from something as basic as git checkout to what we will describe in this article which is private NPM.

For us, the reason for our team choosing a private NPM is that it greatly simplified our deployment process. When deploying our production applications to Bluemix, we previously had to package all of our dependencies as part of the build process, package them into one single deployment artifact and deploy the entire solution to Bluemix. This significantly increased our build time since we needed to download all of our dependencies (public and private) to our build machine and then had to package all of these dependencies into a single compressed file to deploy. This resulted in substantially larger node applications that would frequently timeout during the push of these applications to Bluemix. Since we had approximately 30 applications we were managing in 3 Bluemix Regions, this really impacted our turnaround time for deployments. We were also unable to leverage one of the key features of Bluemix and Cloud Foundry Buildpacks where a Cloud Foundry push would automatically `npm install` all of the packaged defined in your package.json. We knew there had to be a better way.

There were many benefits seen almost immediately by moving to private NPM. The first was that we significantly reduced our deployment times since the deployment process was only pushing the resources that were specific to the project and not their dependencies. This really helped us promote industry best practices that we developed around micro services and delivering small sets of reusable libraries (modules) across all of our micro services such as logging, health checks, security, Cloudant and Elasticsearch. Private NPM also natively supported versioning of modules which allowed us to develop, manage and deploy many different versions of our modules concurrently and provided clear migration paths of modules by leveraging semantic versioning of our resources. Last and not least, we were able to leverage built in security that is provided with NPM to restrict access to a subset of our developer population and build systems that had the appropriate security token. This made it very easy to denote who can read and write to our repository which is important for maintaining reputable dependencies as well as governance control during deployment.

What is Sinopia?

Now that we had determined that we wanted to leverage a private NPM repository, the next question that came up was actually two questions. The first one was what do we use and is there one already available for us to use within IBM with public access to our Bluemix applications. The answer quickly became Sinopia, as there was no solution that existed inside of IBM that satisfied our requirements for Bluemix.

Sinopia was selected initially since it is an open source project with a rather vibrant user community (29 committers, 65 releases and 6K downloads per month). In addition, IBM has already approved the usage of Sinopia and this selection was validated by one of our recent acquisitions (StrongLoop) that leverages Sinopia in their devOps pipeline and internally in many other different manners.

The part we liked best about Sinopia is that it solved our immediate problem as it is designed to host private NPM modules, it integrated with our build pipeline and we were able to add it to our deployment process relatively painlessly. With a small change to our npmrc file to point to our private NPM repository, the deployment process on Bluemix installed our private node modules as part of the `push` to Bluemix.

But wait.. it gets better. As Sinopia is written in Node.js, we were able to easily host this as a Bluemix application as well as run it locally for development purposes without having to stand up many dependencies. With their simplified security model (and various community modules to handle pluggable security models such as ldap), we were able to quickly define Roles and ACLs. This really helped the team get this up and running in a very small amount of time.

The end result is that we have a highly scalable, shadow of the public NPM that cached all of our public NPM modules in addition to our private node modules that we used for our applications.

Deploy Sinopia to Bluemix

Given Sinopia’s packaging, its easiest to create a small Node project that wraps Sinopia as a dependency in its package.json and run it on start:

{

    "name": sinopia-server,
    "version": "1.0.0",
    "description": sinopia-server wrapper",
    "scripts": {
        "start": "node node_modules/.bin/sinopia -l ${VCAP_APP_HOST:=localhost}:${VCAP_APP_PORT:=4873} -c ./config.yaml",
        "test": "echo "Error: no test specified" && exit 1"
    },
    "author": "",
    "license": "ISC",
    "dependencies": {
        "sinopia": ">= 1.x
    }
}

The important piece here is the bash style variable substitution that either sources proper environment variables while running in Bluemix, or uses a set of fallback configurations for local instances.

Before you deploy to Bluemix, youll want to update the config.yaml to make sense for you. A simple setup would include a dev user and an admin user, one with the ability to get all modules, and one with the ability to publish all modules.

# Look here for more config file examples:
# https://github.com/rlidwka/sinopia/tree/master/conf
#

# path to a directory with all packages
storage: ./storage

users:
  dev:
    password: <encrypted password>
  admin:
    password: <encrypted password>

auth:
  htpasswd:
    file: ./htpasswd
    # Maximum amount of users allowed to register, defaults to "+inf".
    # You can set this to -1 to disable registration.
    max_users: -1

# a list of other known repositories we can talk to
uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access:
        - dev
        - admin
    publish: admin
    storage: './private'

  '*':
    # allow all users (including non-authenticated users) to read and
    # publish all packages
    #
    # you can specify usernames/groupnames (depending on your auth plugin)
    # and three keywords: "$all", "$anonymous", "$authenticated"
    access:
        - dev
        - admin

    # allow all known users to publish packages
    # (anyone can register by default, remember?)
    publish: admin

    # if package is not available locally, proxy requests to 'npmjs' registry
    proxy: npmjs
    storage: './public'


# log settings
logs:
  - {type: stdout, format: pretty, level: http}

# Webhook settings
webhook:
  url: <webhook>

Additionally, you can add a webhook to announce changes to your registry via platforms such as Slack.

You should be ready to

cf push

from your Sinopia-server project!

Shameless Plug

Sinopia utilizes the file system for storing and managing the packages in its registry. Since Bluemix does not offer disk persistence for applications, it was important for us to find another means of persistence. The solution was to fork the project and implement a CouchDB backend that would replace the file system backed storage. We chose CouchDB because we could then use Cloudant when deployed to Bluemix.

After the work was done on our fork, we were left with a stateless front end that could be scaled independently of whatever Couch/Cloudant database was backing it. Binding a Cloudant service instance to our Bluemix Sinopia-server project then provided us the persistence that we required in order to withstand crashes and allow us to scale the front-end independent of the packages stored in the registry.

We are currently in the process of externalizing our fork and getting the functionality merged back into the Sinopia project. Unfortunately the maintainer is not active, but there are community efforts to recenter around another Sinopia project due to its popularity and high demand. You can find our fork and see the work being done on it here.

Originally published at developer.ibm.com.