Running your own ILP connector

Add redundancy and liquidity to the ILP network

This post was inspired by the huge growth of the Codius network and the fact that the hundreds of payment channels now open to facilitate this has put a big load on those of us running the core nodes on the Interledger network.

To track the growth of the Codius host network, check out the awesome codiushosts.com

Every Codius host runs moneyd, a mini ILP connector that is optimised to have a single uplink to the Interledger and then accept multiple incoming connections from containers running on the host. This gives containers access to ILP so they can both send and receive payments.

As part of setting up a Codius host, operators configure their moneyd uplink and have a choice of only a small number of Interledger Service Providers (ILSPs) to connect to. An ILSP is a connector that accepts unsolicited incoming peering requests. It will have multiple child nodes connect to it and will assign them each an address and then route ILP packets back and forth for them onto the network. (It’s modelled on the idea of an ISP that provides customers access to the Internet)

Having only a small group of ILSPs is bad, both for redundancy and decentralisation, but also because the hosts of those ILSPs need to commit capital to a payment channel for each client. I’ve got almost 1000 XRP locked up at the moment.

So, the network needs to grow, and this post is a basic guide to a running your own ILSP node and being a part of this growth.

We’re also working hard on bringing Lightning support to the network soon so there is likely to be a pretty lucrative opportunity, as a connector between XRP and BTC when that happens, so this is not just for goodwill either.

But, for now, let’s just add some XRP liquidity and some network redundancy and be prepared for the additional currencies that are coming soon.

The Basics

A connector or node on the ILP network is a running instance of ilp-connector (it could be another implementation of an ILP node although none exist yet that we know of).

You can deploy it any number of ways as long as it can connect to peers, forward ILP packets and exchange routing information. These instructions explain how I have deployed my connector but you could find alternative ways to do it, and if you do, please share them with the community in our Gitter channel.

I run ilp-connector (written in TypeScript and so run using NodeJS) using PM2 as a way to “daemonize” the process (so it does things like survive restarts) and also because it allows me to do some basic remote monitoring.

Using PM2 is also the way we configure our Terraform AWS installer. That’s another way to run your connector if you’re keen to use AWS and get some free automation of the whole setup. I am busy migrating my connector from AWS after using the Terraform deployment first because I want it physically located in Africa. These instructions come from deploying another small connector at DigitalOcean to use for some testing.

With the connector, you’ll need to configure at least two plugins. Plugins are responsible for the connections you have with other nodes. The two you need to get started are:

ilp-plugin-xrp-asym-server: is the plugin that exposes the ILSP server to downstream clients. It accepts new connections and creates an account internally for each. You’ll expose this service publicly and “customers” will configure their money instances to connect to this.

moneyd’s xrp-uplink plugin is actually a wrapper around ilp-plugin-xrp-asym-client.

ilp-plugin-xrp-paychan: is a plugin for establishing a direct peering relationship with another connector/node over XRP. This is the one you’ll use to peer with other connectors directly.

Links

Rather than repeat a lot of guides that are already out there I have put pointers to these to get many of the pre-requisites setup on your server. All links to external steps are written as headings (the ones that are links are underlined) so you can skim through and find them easily (if they are not links then follow the instructions on this page).

I have setup a server running the latest Ubuntu 18 LTS. Depending on your host you may have this available or you can pick an older version. Our core dependencies are Nginx, Redis and NodeJS so it shouldn’t matter much.

STEP 1 → Setup and harden your server

This handy script will do everything described in the steps above and save you some time. If it doesn’t work for you, go back and follow the instructions.

With your server ready to go, we need some prerequisite software.

The first thing we need is Nginx. It will sit in front of the connector software and handle TLS so all external connections are secured and simply proxied inwards to our connector service and plugins.

Before you start you’ll need to register a domain name to use for your connector and you’ll need to setup an A record that points to the server.

I used the domain ilp.hopebailie.com so if I run dig I get the following which tells me that there is an A record pointing to my server at 159.89.3.52:

$ dig ilp.hopebailie.com +short
159.89.3.52

You need to make sure this is setup BEFORE you try getting certificates, and may have to wait for the DNS to propagate if you’re using an existing domain name that already had a different A or CNAME record.

Now follow the instructions in step 2 to install Nginx. Remember to substitute your own domain name for example.com in all the instructions. Also, you don’t really need the www sub-domain so you can leave that out of the server block configuration if you like.

STEP 2 → Install Nginx

Now you have a basic web server setup. You should have visited your domain and seen the test page you created during the process that looked something like this:

Next step is to secure it with some free SSL certificates from LetsEncrypt.

STEP 3→ Get SSL certs from LetsEncrypt

If you followed the instructions above, you have Nginx setup to serve web pages securely. If you visit your domain you should be shown the same test page you created in Step 2 even if you visit your server using https.

STEP 4 → Setup reverse proxy and disable port 80

If you followed the previous steps verbatim then you should have a folder on your server with your server-block nginx config and a file named after your domain. Example:

$ ls -al /etc/nginx/sites-available
total 16
drwxr-xr-x 2 root root 4096 Jul 18 14:31 .
drwxr-xr-x 8 root root 4096 Jul 18 14:33 ..
-rw-r--r-- 1 root root 2416 Apr  6 06:31 default
-rw-r--r-- 1 root root  733 Jul 18 14:32 ilp.hopebailie.com

First delete default (we don’t want to serve any web pages) and then edit ilp.hopebailie.com as follows:

1. Remove the listen directives for port 80
2. Remove the root directive
3. Remove the index directive
4. Remove the server_name directive (assuming you won’t have other services proxying through nginx)
5. Replace the location directive with:

location / {
    proxy_pass http://127.0.0.1:7$server_port;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}
location ~ /.well-known {
    allow all;
}

This redirects everything on an external port to the same internal port but prefixed with a 7. This means we can open ports using the firewall and know which internal port they will be proxied to.

By default, wss (secure WebSockets requests) will be proxied to 7443, so on my server connections to wss://ilp.hopebailie.com are proxied to ws://127.0.0.1:7443 so I’ll configure my ILSP plugin to listen on that port.

This also puts in a directive that allows LetsEncrypt’s certbot to continue renewing our certificates even though we are locking down the server.

5. Add this block to the file (outside the server block).

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

Finally, you should have a file that looks something like this:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}
server {
    location / {
         proxy_pass http://127.0.0.1:7$server_port;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection $connection_upgrade;
    }
    location ~ /.well-known {
        allow all;
    }
    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/ilp.hopebailie.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ilp.hopebailie.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Test your new config:

$sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Reload Nginx:

$ sudo systemctl reload nginx

Close port 80 on the firewall:

$ sudo ufw status numbered

Shows a list of rules, likely something like this:

Status: active
To                         Action      From
--                         ------      ----
[ 1] OpenSSH                    ALLOW IN    Anywhere
[ 2] Nginx Full                 ALLOW IN    Anywhere
[ 3] OpenSSH (v6)               ALLOW IN    Anywhere (v6)
[ 4] Nginx Full (v6)            ALLOW IN    Anywhere (v6)

Delete the Nginx Full rules and then allow Nginx HTTPS :

$ sudo ufw delete 4
$ sudo ufw delete 2
$ sudo ufw allow 'Nginx HTTPS'

Re-run the certbot renewal to test it will still work:

$ sudo certbot renew — dry-run

If you visit your server now via a browser now you’ll get a “504: Bad Gateway” error from nginx because we have nothing listening internally on port 7443.

You should see this error in your logs, a handy place to look if you’re troubleshooting later:

$ sudo tail /var/log/nginx/error.log

STEP 5 → Install NVM

With nvm installed you can easily install the latest LTS of node, if you haven’t already:

$ nvm install --lts

STEP 6 → Test WebSockets

We can use wscat to run a little WebSocket server for a few seconds while we test that our firewall is working.

$ npm install -g wscat
$ wscat -l 7443

Now visit your server in a browser again and you should see:

Upgrade Required

This means that at least your browser and wscat started the handshake to open a WebSocket connection so it looks like things are setup correctly.

You can go a step further and install wscat on another machine and test the connection by running:

$wscat -c wss://ilp.hopebailie.com --slash
/ping
< Received pong

Step 7 → Install Redis

(skip steps 4 and 5 as you’ll only use Redis locally)

Redis is used by plugins to persist their balances. It’s an essential component for your ILSP as it is used by ilp-plugin-asym-server to record the state of all your incoming connections.

ilp-connector supports pluggable storage adaptors so we’re going to use Redis.

On my connector I configured Redis to use a local socket instead of TCP for performance. That’s a little advance for this guide but worth considering if you are seeing high volume.

IMPORTANT: You’ll need to configure Redis so that it can survive service restarts and down-time. Read about how Redis persists its data to disk here and ensure you’re comfortable that your system will survive a failure and that you are making backups.

Step 8 → Install PM2

$ npm install -g pm2
$ pm2 completion install
$ pm2 startup
[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/home/ilp/.nvm/versions/node/v10.6.0/bin /home/ilp/.nvm/versions/node/v10.6.0/lib/node_modules/pm2/bin/pm2 startup systemd -u ilp --hp /home/ilp

Depending on your operating system PM2 may output a command that you need to run to finish the service installation. Note that I had to copy and paste the last line of output above.

Step 9→ Install ilp-connector and plugins

Pick a directory where you want to run the connector. I’m simply using my user’s home directory but there are probably better long term options like /srv. Make sure your user has permission to write to the folder (ideally it should own it).

In that directory clone the connector and install it:

$ git clone https://github.com/interledgerjs/ilp-connector
$ cd ilp-connector
$ npm install

Now install the additional plugins you need:

$ npm install ilp-plugin-xrp-asym-server ilp-plugin-xrp-paychan ilp-store-redis

Step 10 → Get an XRP Ledger address

Your connector needs a Ripple address and it will also need some XRP to meet the reserve requirements (that will go up quickly as people join your connector because of the channels your connector will create with each client).

An easy tool to create new Ripple addresses is ripple-wallet-cli which you can install on the server and run as follows:

$ npm install -g ripple-wallet-cli
$ ripple-wallet-cli generate

It has useful functions you may want to use in future like checking balances and making payments too.

Record the address and secret generated by the wallet tool.

IMPORTANT: Anyone with that secret can steal your money. It’s a VERY sensitive value. This is what is often termed a “hot wallet”. You should only keep as much XRP in that account as needed for your connector to run. If you have a lot of excess XRP in that account move some of it to a more secure “cold wallet”.

I like this explanation of hot and cold wallets from Leah Stella Stephens.

You’ll need to put some XRP into this account before it will be usable (at least 20 XRP to meet the minimum reserve and then another 20 or so to cover the extra-reserve requirements of opening channels). The reserve requirement will go up fast if you have a lot of clients on your ILSP so be prepared to keep this topped-up. The good news is it will also drop down when they disconnect and the channel closes.

Step 11 → Find a peer that is already on the network

You’re joining a network so you need to find at least 1 peer that will connect with you and give you access to the rest of the network. If you’re looking for one, try reaching out on the Gitter chat.

Once you have a peer lined up, you’ll need to agree on who will host the WebSocket server and who will be the client. This is just for connection establishment, you’ll both establish payment channels once you are connected. Also note that this is orthogonal to a parent/child relationship between peers, in this case we just need to decide who is listening for a WebSocket connection and who is connecting. Once the connection is established either party could be a parent, child or peer.

In the config below I’m assuming we’re going to be a client as this is simpler and requires no firewall changes. If you set your ilp-plugin-xrp-paychan instance to be a server then you’ll need to open a port for it.

Remember that if you open port 444 then this is the value you must give your peer but you will configure your plugin to listen on port 7444 internally.

You also need to agree on a secret that you’ll both use to secure the connection (just get a base64 encoded random 32 byte number):

$ cat /dev/urandom | head -c 32 | base64

The details on where to use this is shown in the config below.

Step 12 → Pick an ILP Address

You need to pick an address in the global address space at which to advertise your connector. ILP has no registry so it’s up to you to pick a unique name. If there are conflicts then you’ll get a lot of payments you can’t fulfil and likewise, a lot of payments that are supposed to go to you will not get there and will also go unfulfilled.

There’s no financial risk but it will mean a lot of noise and errors on the network.

The address I am using is g.hopebailie. You should pick a similar, top-level, address or your peer may assign you one that exists in their own address space.

To learn more about addresses you can read the spec here.

Step 13 → Create your config file

You can use PM2 to create an initial config file (or simply use the example below):

$ pm2 init

If you use the default PM2 -generated file, you’ll want to:

1. Remove the deployment section. We won’t use that for now.
2. Add two constants that hold your Ripple address and secret so you can re-use these for any plugins that are configured to use XRP ledger.
3. Define the two plugin configurations for the ilp-plugin-xrp-paychan and ilp-plugin-xrp-asym-server. There are details on the Github README’s but simply using the example below should suffice.
4. Define connection details for the ilp-store-redis adapter (the example below should work as-is).

You should end up with a file like this:

'use strict'
const path = require('path')
const address = '<YOUR RIPPLE ADDRESS>'
const secret = '<YOUR RIPPLE SECRET>'
const peer1 = {
    relation: 'peer',
    plugin: 'ilp-plugin-xrp-paychan',
    assetCode: 'XRP',
    assetScale: 9,
    balance: {
        maximum: '1000000000',
        settleThreshold: '-5000000000',
        settleTo: '0'
    },
    options: {
        server: <PEER BTP URL>
        rippledServer: 'wss://s2.ripple.com',
        peerAddress: '<PEER RIPPLE ADDRESS>',
        address,
        secret
     }
}
const ilspServer = {
    relation: 'child',
    plugin: 'ilp-plugin-xrp-asym-server',
    assetCode: 'XRP',
    assetScale: 6,
    options: {
        port: 7443,
        xrpServer: 'wss://s2.ripple.com',
        address,
        secret
    }
}
const connectorApp = {
    name: 'connector',
    env: {
        DEBUG: 'ilp*,connector*',
        CONNECTOR_ENV: 'production',
        CONNECTOR_ADMIN_API: true,
        CONNECTOR_ADMIN_API_PORT: 7700,
        CONNECTOR_ILP_ADDRESS: '<YOUR ILP ADDRESS>',
        CONNECTOR_BACKEND: 'one-to-one',
        CONNECTOR_SPREAD: '0',
        CONNECTOR_STORE: 'ilp-store-redis',
        CONNECTOR_STORE_CONFIG: JSON.stringify({
            prefix: 'connector',
            port: 6379
        }),
        CONNECTOR_ACCOUNTS: JSON.stringify({
            <NAME of PEER>: peer1,
            ilsp: ilspServer
        })
    },
    script: path.resolve(__dirname, 'src/index.js')
}
module.exports = { apps: [ connectorApp ] }

Step 14 → Start

If you used PM2 to generate the file it will be called ecosystem.config.js. You can now start the connector:

$ pm2 start ecosystem.config.js

Use PM2 to view the logs or monitor the service

$ pm2 monit connector
$ pm2 logs connector

If you want PM2 to restart the same services on a restart them dump the current config using

$ pm2 save

Next Steps

That covered the basics of getting a connector up and running. In my next blog post I’ll describe some the tools you can use to monitor and tune your connector and add some troubleshooting tips.

UPDATE: I put a few tips ‘n tricks in this recent blog post

If you are having any issues let me know via Twitter or on the Gitter chat.