Run Native Binaries in Computes

Photo by Ash Edmonds on Unsplash

Thanks to the new whitelist and manifest features of Computes v14.0.0, we can now allow native binaries to run in your network. This allows you to create binaries that can access features that Docker may not offer, especially access to the GPU of the nodes in your Computes network.

This post will show you how to set up a manifest, whitelist, and binaries that can run as tasks on many different operating systems.

Binaries for Every Platform

Thanks to the Computes manifest, you can specify different binaries for each platform we support. We’ve created an example application to help demonstrate.

Build the example app

bin-echo uses goreleaser to cross-compile our application for many different platforms. If you’d like to try for yourself, you can build locally using the following command.

goreleaser release --skip-publish --rm-dist --skip-validate

This will create a dist/ directory with the binary wrapped in an archive.

Upload binaries to IPFS

First, let’s upload each of these binaries to IPFS. The Computes node will use IPFS to download the appropriate binary archive and extract the binary. We support many different archive formats, including tar.gz, zip, tar.bz2, rar, and many more. Using archives allows you to include any necessary dlls, assets, or other files needed for your executable to run.

⟩ ipfs add bin-echo-v1.0.0-* -w
added QmYYScH89Jwif7cpsE7SLb77oQF8Xue1o5HyDKBQeaYoYk bin-echo-v1.0.0-darwin-amd64.zip
added QmYxEnsPCEVsgAMQL9Ny4aMHEGZ9rBhNyBHX5SkND27Ns3 bin-echo-v1.0.0-linux-386.tar.gz
added QmQqnijZGXY6bNHMk159327oQKJy4Ziw7ugUmwWw9w1tDJ bin-echo-v1.0.0-linux-amd64.tar.gz
added QmRFGEu31iYmkwG73Df9htRkmaYG9syk4ChsLCZvvWixEE bin-echo-v1.0.0-linux-armv7.tar.gz
added QmZT9t2xpyHhDMAe479t5VbFX9bv9oFshdZ8dCnLdDbQCm bin-echo-v1.0.0-windows-386.zip
added Qmao7w4wmQYD7qHHR23HGNnPY5gWB7vcHMoZ1tqYAwPCSR bin-echo-v1.0.0-windows-amd64.zip
added QmatbWWsCucVsJT8HVhsjsj4JnSaGceS44EgxcSaSUXP8p

Create the Manifest

The manifest will specify which binary archive to download for each operating system + architecture combination.

{
  "darwin-amd64": {
    "file": {
      "/": "QmYYScH89Jwif7cpsE7SLb77oQF8Xue1o5HyDKBQeaYoYk"
    },
    "executable": "bin-echo",
    "format": "zip"
  },
  "linux-amd64": {
    "file": {
      "/": "QmQqnijZGXY6bNHMk159327oQKJy4Ziw7ugUmwWw9w1tDJ"
    },
    "executable": "bin-echo",
    "format": "tar.gz"
  },
  "linux-386": {
    "file": {
      "/": "QmYxEnsPCEVsgAMQL9Ny4aMHEGZ9rBhNyBHX5SkND27Ns3"
    },
    "executable": "bin-echo",
    "format": "tar.gz"
  },
  "linux-armv7": {
    "file": {
      "/": "QmRFGEu31iYmkwG73Df9htRkmaYG9syk4ChsLCZvvWixEE"
    },
    "executable": "bin-echo",
    "format": "tar.gz"
  },
  "windows-386": {
    "file": {
      "/": "QmZT9t2xpyHhDMAe479t5VbFX9bv9oFshdZ8dCnLdDbQCm"
    },
    "executable": "bin-echo.exe",
    "format": "zip"
  },
  "windows-amd64": {
    "file": {
      "/": "Qmao7w4wmQYD7qHHR23HGNnPY5gWB7vcHMoZ1tqYAwPCSR"
    },
    "executable": "bin-echo.exe",
    "format": "zip"
  }
}

file is an IPLD link to the archive containing the binary.

executable is the name of the executable contained inside the archive.

format is the archive format.

Add Manifest to IPFS

cat manifest.json | ipfs dag put
> zdpuAudrVQU58VdxUrZN2irD8qbxgRWqa4ra2DXcAxQRiwfkq

Create Whitelist

{
  "runners": {
    "bin-json-runner": {
      "manifests": [
        { "/": "zdpuAudrVQU58VdxUrZN2irD8qbxgRWqa4ra2DXcAxQRiwfkq" }
      ]
    }
  }
}

Add Manifest to Whitelist

cat whitelist.json | ipfs dag put
> zdpuAp1FMgjiJ8b5MWVADxQDhx9w66K152972Mw6hH11kaH2x

Example Task

{
  "input": {
    "dataset": 2
  },
  "taskDefinition": {
    "runner": {
      "type": "bin-json-runner",
      "manifest": { "/": "zdpuAp1FMgjiJ8b5MWVADxQDhx9w66K152972Mw6hH11kaH2x" }
    },
    "result": {
      "action": "set",
      "destination": {
        "dataset": { "/": "<dataset hash here>" },
        "path": "echo/results"
      }
    }
  },
  "status": {
    "/": "<status hash here>"
  }
}

Running this task will check the manifest, download the archive for the os+architecture, unpack it, and execute the binary listed in the manifest description.

What’s Next?

Computes will soon use process isolation, chroot, jail, or whatever security features our supported operating systems will allow us to use to keep your nodes safe and secure.