Connector operations

A few tips and tricks for operating your ILP connector

Visualize the network using moneyd-gui

A few weeks ago I published a short guide on how to setup a connector on the Interledger and since then we’ve had new connectors join the network almost daily.

We recently started a new Gitter chat specifically for connector operators so please join us there if you’re running (or planning to run) a connector.

In this post I want to share a few things I have been playing with on my connector that help me add new peers easily and manage my connector. There are a few major changes coming to the software that will make operations a lot easier but for now here’s a few handy hacks.

Tip #1 Put your Ripple address and secret into env variables

Rather than having the secret for your connector’s XRP address floating around in files that you may mistakenly share when you’re looking for help, load them into env variables so they are only accessible to the account under under which your connector runs.

My connector runs under the account ilp so I have added the following few lines to the .bashrc file in the root of that user’s home directory. (Different operating systems and distros may have other places to setup an environment variable so YMMV).

### Setup XRP address & secret
export XRP_SECRET="sgfoaeyrgfoerygfiwerygfeiyrfg"
export XRP_ADDRESS="riuherfpiup98hergpuewrpigup9erughpiquerg"

Once that file has been saved you can either re-load your profile or just log out and back in to test it

$echo $XRP_ADDRESS
riuherfpiup98hergpuewrpigup9erughpiquerg

Now you can access these values from within any NodeJS process run under that user by simply accessing the property process.env.XRP_ADDRESS

SECURITY NOTE: This is not necessarily any more secure than storing this value in a config file somewhere with appropriate permissions. This is just a way to ensure it is only stored in one place on disk and in a place you are unlikely to inadvertently share with others when exchanging configs etc. I would not recommend holding more XRP in the account your connector uses than is necessary and if you are earning revenue into that account, withdraw some regularly to ensure that if the server is compromised you only lose what is in that account at that time

Tip #2 Put your config into /etc/ilp-connector

The great thing about using PM2 to load our connector is that the config file we load up is just a Javascript file and we can add whatever logic we want in there to output our config to PM2.

I’ve followed the linux convention and moved my config out of the root of the ilp-connector directory so it won’t conflict with code updates, and into /etc/ilp-connector

Create the directory and give the account under which you run your connector ownership. (Read-access may be sufficient here, I’ll leave it to the Linux experts to suggest a better approach if this is not it)

$ sudo mkdir /etc/ilp-connector
$ sudo chown ilp:ilp /etc/ilp-connector/

Now copy your ecosystem.config.js file into this folder.

In it’s place I have a simple file in my ilp user’s homedir which PM2 will launch by default whenever you call pm2 start --update-env:

module.exports = {
  apps : [
    Object.assign(
      require('/etc/ilp-connector/ilp-connector.conf.js'),
      { name: 'ilp-connector' }
    )
  ]
};

In future I can add multiple apps to this config and control them individually through the PM2 command-line application.

I’ve renamed my root config file to ilp-connector.conf.js to try and establish/align with some conventions I’ve noticed among other services. It no longer exports a full PM2 ecosystem config but rather just the config for the ilp-connector app (details below).

Tip #3 Split your config into separate files

I started to find adding new peers a very unwieldy process so I have made some tweaks to the root config file that allow me to borrow the config pattern from nginx which I really like. That is, to have a stand-alone file for each peer and to enable/disable them simply through creating symbolic links to the config.

I started by creating the following two directories:

$ mkdir /etc/ilp-connector/peers-available
$ mkdir /etc/ilp-connector/peers-enabled

Then I put the config for each peer into a file in peers-available and to enable them I simply symlink that config into peers-enabled. (If you have used httpd or nginx this will feel very familiar)

A typical peer config file looks like this:

module.exports = {
  relation: 'peer',
  plugin: 'ilp-plugin-xrp-paychan',
  assetCode: 'XRP',
  assetScale: 9,
  balance: {
    maximum: '1000000000',
    settleThreshold: '-1000000',
    settleTo: '0'
  },
  options: {
    listener: {
      port: 7444,
      secret: 'verysecretvaluethatnobodycanguess'
    },
    rippledServer: 'wss://s2.ripple.com',
    assetScale: 9,
    address: process.env.XRP_ADDRESS,
    secret: process.env.XRP_SECRET
  }
}

That is saved with the filename peer1.conf.js which means it will be loaded into the config under the account name peer1.

IMPORTANT: If you are moving to this system from a single config make sure you don’t change the name of the accounts you have configured as this will mean you lose all the stored state in the DB for that account.

Now to enable this peer I simply create a link in peers-enabled like this:

$ ln -sr /etc/ilp-connector/peers-available/peer1.conf.js /etc/ilp-connector/peers-enabled/peer1.conf.js

And to disable it I just delete the link (while preserving the config):

$ rm /etc/ilp-connector/peers-enabled/peer1.conf.js

(You could put these commands in a user script to make them easy to do when you’re in a rush, I have put them into an npm module called ‘ilp-connector-config’ and @n3tc4t has also got his script that does the same: https://github.com/N3TC4T/ilp-peer-tools)

I have created XRP peer templates in the peers-available folder for both server and client configs and I can simply copy these and tweak them to add new XRP paychan peers.

I also split any of the config that I consider unique per deployment into stand-alone files, so I have a node.conf.js that looks like this:

module.exports = {
  address: 'g.hopebailie',
  backend: 'one-to-one',
  spread: 0,
  apiPort: 7769
}

And a store.conf.js that looks like this:

module.exports = {
  plugin: 'ilp-store-redis',
  config: {
     prefix: 'connector',
     path: '/var/run/redis/redis-server.sock'
  }
}

I like this because as I write tools to work with the connector I may want to pull these files in to load specific config, for example I am working on a tool to dump the data from the store that doesn’t need the whole connector config to work.

Tip #4 Test your config before you reload the connector

One of the features of nginx that I love is the ability to run nginx -t to quickly test your config before attempting a reload. It’s not guaranteed to catch issues but it prevents you reloading your service and then killing it because you have silly errors in your config which you need to fix frantically while your service is down.

Because our config is an executable javascript file I’ve littered it with checks and you can simply run it through NodeJS to evaluate your config.

My ilp-connector.conf.js now looks like this:

'use strict'
const DEBUG = '*'
const fs = require('fs')
const path = require('path')
function isConfigFile (file) {
  return file.endsWith('conf.js')
}
function getAccountFromConfigFileName (file) {
  return file.slice(0, file.length - 8)
}
function loadNode (confPath, appConfig) {
  console.log(`Loading node config from ${confPath}`)
  const node = require(confPath)
  if (node.address) {
    appConfig.env['CONNECTOR_ILP_ADDRESS'] = node.address
  }
  if (node.backend) {
    appConfig.env['CONNECTOR_BACKEND'] = node.backend
    console.log(` - Backend: ${node.backend}`)
  } else {
    throw new Error('A rates backend must be must be provided for this node')
  }
  if (node.spread) {
    appConfig.env['CONNECTOR_SPREAD'] = '' + node.spread
    console.log(` - Spread: ${node.spread}`)
  } else {
    appConfig.env['CONNECTOR_SPREAD'] = '0'
    console.log(' - No spread provided, defaulting to zero.')
  }
 if (node.apiPort) {
   console.log(` - Api enabled on port ${node.apiPort}`)
   appConfig.env['CONNECTOR_ADMIN_API'] = 'true'
   appConfig.env['CONNECTOR_ADMIN_API_PORT'] = '' + node.apiPort
 }
}
function loadStore (confPath, appConfig) {
  console.log(`Loading store config from ${confPath}`)
  const store = require(confPath)
  if (!store.plugin) {
    throw new Error('No store plugin configured')
  } 
  appConfig.env['CONNECTOR_STORE'] = store.plugin
  appConfig.env['CONNECTOR_STORE_CONFIG'] = JSON.stringify(store.config)
  console.log(`- Using ${store.plugin} plugin`)
}
function loadAccounts (peerConfDir, appConfig) {
  console.log(`Loading account config from ${peerConfDir}`)
  const accounts = {}
  const peers = fs.readdirSync(peerConfDir).filter(isConfigFile)
  if (!peers.length) throw new Error('No peer configurations found')
  peers.forEach((file) => {
    const account = getAccountFromConfigFileName(file)
    accounts[account] = require(peerConfDir + '/' + file)
    console.log(`- ${account} (${accounts[account].relation}) : ${accounts[account].plugin}`)
  })
  appConfig.env['CONNECTOR_ACCOUNTS'] = JSON.stringify(accounts)
}
const connectorApp = {
  name: 'connector',
  env: {
    DEBUG,
    CONNECTOR_ENV: 'production'
  },
  script: path.resolve('/srv/ilp-connector/src/index.js')
}
// Ensure XRP config is provided
if (!process.env.XRP_ADDRESS || !process.env.XRP_SECRET) {
 throw new Error('XRP_ADDRESS and XRP_SECRET must be defined')
}
loadNode(path.resolve(__dirname, './node.conf.js'), connectorApp)
loadStore(path.resolve(__dirname, './store.conf.js'), connectorApp)
loadAccounts(path.resolve(__dirname, './peers-enabled'), connectorApp)
if (!fs.existsSync(connectorApp.script)) {
  throw new Error(`Couldn't find ilp-connector start script at ${module.exports.apps[0].script}`)
}
module.exports = { apps: [ connectorApp ] }

Now I can execute this file to check the config:

$ node /etc/ilp-connector/ilp-connector.conf.js

But I can also pass the same file into PM2 to run my connector:

$ pm2 restart /etc/ilp-connector/ilp-connector.conf.js --update-env

Tip #5 Use a local moneyd-gui to monitor your connector

moneyd-gui is a great front-end for moneyd, but because moneyd is just a connector we can use the same tool on our own connector.

Rather than opening any new ports or running anything extra on your connector you can just use ssh to tunnel into your connector when you need to view the tool.

The one prerequisite is that you have the ilp-plugin-mini-accounts plugin configured as this will be used by moneyd-gui to send and receive ILP packets directly through your connector.

You can use the GUI without it but some functionality will be limited. I’ve used the following peer config on my connector:

module.exports = {
  relation: 'child',
  plugin: 'ilp-plugin-mini-accounts',
  assetCode: 'XRP',
  assetScale: 6,
  options: {
    port: 7768
  }
}

Note that you can use any port but make sure when you tunnel to your connector that you use port 7768 on your local machine (you might have to stop your local moneyd if it’s running and using that port).

Install moneyd-gui on your local machine:

$ npm install -g moneyd-gui

Now tunnel through to your connector and bind the local ports 7768 and 7769 to the mini-accounts plugin and admin API on your connector.

I have setup my local ~/.ssh/config so I don’t need to pass in as many params to ssh. It contains the following config (among others):

Host connector
Hostname <IP ADDRESS OF MY CONNECTOR>
User ilp
Port <SSH PORT, DEFAULT IS 22>
IdentityFile ~/.ssh/connector_rsa

You can find more info about configuring ssh in the manpages or online.

So to SSH into my server I would normally just type:

$ ssh connector

Now, to tunnel those 2 ports I add some extra parameters

$ ssh -N -L 7768:localhost:7768 -L 7769:localhost:7769 connector

This maps my local ports to the same ports on the connector. You can use different remote values as long as you stick with the defaults locally as that’s what moneyd-gui will look for.

Now, in another terminal run:

$ moneyd-gui

You should see some logging output and then a message to say that the server is listening. By default you can then access the GUI at: http://localhost:7770

Wrap-Up

These are just a few tips to help make running your connector easier. These are techniques I have used and you may have some other (probably better) ideas in which case you should come and share them on Gitter.

A skeleton of my /etc/ilp-connector directory structure is available at https://github.com/adrianhopebailie/ilp-connector-config. You can also install it using the npm module:

$ npm install -g ilp-connector-config

UPDATE: I added a basic bash script to the module called ilp-connector-config which supports 4 commands, test, restart, enable and disable

Make sure it has execute permissions and put it in your path.

The following will enable the peer peer1:

$ ilp-connector-config enable peer1

The following will disable it:

$ ilp-connector-config disable peer1

The connector must still be restarted after changes. This can be done with:

$ ilp-connector-config restart

The config can also be tested first with:

$ ilp-connector-config test