Infrakit: HPC Infrastructure

>>> Docker Kata 007

Are you interested in High Performance Computing (HPC)?

If so, read on to see how we’ll use Docker’s InfraKit toolkit to create and manage self-healing infrastructure that can scale to meet today’s demanding web-scale applications. We’ll abstract away the need to rely on one particular cloud service provider (CSP), and build infrastructure for the hybrid cloud world.

In short, get ready to build the next generation foundation for the HPC clusters of tomorrow: Docker Swarms in production here we come (and maybe Mesos or Kubernetes, too!)

In last week’s Docker Kata 006, we took our first look Infrakit. Having played with it a bit over the last week, I wanted to share an early adopter’s impressions with everyone, to see what the software looks like in person.

As previously described, Infrakit is a toolkit for managing infrastructure automation through the use of simple, pluggable components. It allows you to create declarative infrastructure that approximates self-healing through the re-application of desired state through tools such as Terraform and Vagrant. We’ll explore the use of primitives such as groups, instances and flavors here to understand how Infrakit’s plugins are used as primitives in order to interface with cloud or local provider APIs and interfaces.

In today’s Kata, we’re going to focus on building what you can consider “theoretical” infrastructure. We’ll rely on json files and the filesystem to define what a potential infrastructure would look like, and we’ll even be able to scale it up and down and monkey with it, but we’ll avoid concepts that require a connection to a specific cloud provider. There’s enough new stuff here to take things one jump at a time.

Let’s set up a quick golang development environment, and then start testing pieces. Let’s run through each of the steps here. First we’ll install Go from https://golang.org/dl/. Next, verify where Go has been installed.

$ which go
/usr/local/bin/go

I’ve installed Go through Brew, so your path may be somewhere different, such as /usr/local/go or /usr/local/go/bin. The path given by `which go` is yours.

Next, export the go path to your environment $PATH. You can do this by adding this line to your /etc/profile (for a system-wide installation) or $HOME/.profile to make this permanent and automatic. In our case, this command is telling the system to add the go binary directory to the beginning of the system PATH (newpath:$EXISTINGPATH).

$ export PATH=/usr/local/bin/go:$PATH

Now we can create a local Go development environment according to best practices (another good link.) These steps assume that you have a directory in your home folder called ‘go’ that you’ll use to host your different projects.

Create the development folders

$ mkdir -p ~/go
$ export GOPATH=!$
$ export PATH=$GOPATH/bin:$PATH

Once we’ve set up the path, let’s create a folder for the project according to convention, with a ‘src’ directory and project.

$ mkdir -p ~/go/src/github.com/docker
$ cd !$
$ git clone git@github.com:docker/infrakit.git
$ cd infrakit

Install the lint test support.

$ go get -u github.com/golang/lint/golint

Check that you can run some tests

$ WOPR:infrakit jesse$ make ci
+ fmt
+ vet
+ lint
+ vendor-sync
+ coverage
? github.com/docker/infrakit/cmd/cli [no test files]
? github.com/docker/infrakit/cmd/group [no test files]
ok github.com/docker/infrakit/discovery 1.050s coverage: 78.0% of statements
? github.com/docker/infrakit/example/flavor/swarm [no test files]
...

And then we’ll build the binaries for the project, so we can get starting building infrastructure. In essence, we’re creating a bunch of executables that are going to be placed into the ./build directory.

$ make binaries
+ clean
rm -rf build
mkdir -p build
+ binaries
go build -o build/infrakit -ldflags “-X github.com/docker/infrakit/cli.Version=27a1c7a -X github.com/docker/infrakit/cli.Revision=27a1c7a40327b5c3cd8e8b498cd18c8f38337d8c” github.com/docker/infrakit/cmd/cli
go build -o build/infrakit-group-default -ldflags “-X github.com/docker/infrakit/cli.Version=27a1c7a -X github.com/docker/infrakit/cli.Revision=27a1c7a40327b5c3cd8e8b498cd18c8f38337d8c” github.com/docker/infrakit/cmd/group
go build -o build/infrakit-flavor-combo -ldflags “-X github.com/docker/infrakit/cli.Version=27a1c7a -X github.com/docker/infrakit/cli.Revision=27a1c7a40327b5c3cd8e8b498cd18c8f38337d8c” github.com/docker/infrakit/example/flavor/combo
go build -o build/infrakit-flavor-swarm -ldflags “-X github.com/docker/infrakit/cli.Version=27a1c7a -X github.com/docker/infrakit/cli.Revision=27a1c7a40327b5c3cd8e8b498cd18c8f38337d8c” github.com/docker/infrakit/example/flavor/swarm
...

As a side note, the InfraKit repository has been making changes over the past few weeks as would be expected with brand new technology. Make sure that you’re up to date with the latest master if you haven’t updated recently.

$ git fetch -a && git pull origin master

We spent most of our time in Docker Kata 005 examining the purpose and design of InfraKit, but now that we have our software ready to go, let’s start building things!

First off we can test our installation by using the File instance plugin, which can accept any configuration that is then written to disk. This is mostly useful for testing and debugging.

$ build/infrakit-instance-file
INFO[0000] Listening at: /Users/jesse/.infrakit/plugins/instance-file

We’ll also use plugin discovery here to keep track of what is running. These plugins communicate via HTTP and discover each other by looking for socket files in a common directory. The default directory for these plugins is ‘~/.infrakit/plugins’.

Start a Group plugin.

$ build/infrakit-group-default
INFO[0000] Listening at: /Users/jesse/.infrakit/plugins/group

And check it via plugin discovery.

$ build/infrakit plugin ls
Plugins:
NAME LISTEN
group /Users/jesse/.infrakit/plugins/group
instance-file /Users/jesse/.infrakit/plugins/instance-file

Let’s start another Instance plugin, and tell the Instance plugin to store the instances in the tutorials directory as they’re provisioned. We’ll need to give the plugin a different name, since we’re already using the default Instance File plugin above.

$ build/infrakit-instance-file — name=tutorial-instance-file — dir ./tutorial/

If we run plugin discovery, what do you think we’ll see? That’s right, all of our plugins, listening on 3 different unix sockets. Fun! If you check the tutorial directory, you’ll see that it’s empty because we haven’t created any virtual resources yet.

$ build/infrakit plugin ls
Plugins:
NAME LISTEN
group /Users/jesse/.infrakit/plugins/group
instance-file /Users/jesse/.infrakit/plugins/instance-file
tutorial-instance-file /Users/jesse/.infrakit/plugins/tutorial-instance-file

Let’s start one of the provided flavor plugins called ‘vanilla’, which we’ll use to create a group.

$ build/infrakit-flavor-vanilla

Using a combination of our file plugin tutorial-instance-file and the flavor-vanilla plugin, we can create a configuration to feed to the Group plugin.

Copy this whole command into your command line to create the cattle.son file. The cat utility with the EOF is a fancy way of inputting the text between EOF > and EOF into the named file.

$ cat << EOF > cattle.json
{
  "ID": "cattle",
  "Properties": {
    "Allocation": {
      "Size": 5
    },
    "Instance": {
      "Plugin": "tutorial-instance-file",
      "Properties": {
        "Note": "Instance properties version 1.0"
      }
    },
    "Flavor": {
      "Plugin": "flavor-vanilla",
      "Properties": {
        "Init": [
          "docker pull nginx:alpine",
          "docker run -d -p 80:80 nginx-alpine"
        ],
        "Tags": {
          "tier": "web",
          "project": "infrakit"
        }
      }
    }
  }
}
EOF

Since we haven’t built anything yet, we can check to make sure that there are no instances yet created

$ build/infrakit instance — name tutorial-instance-file describe
ID LOGICAL TAGS

We can instruct the group plugin to watch our group, wherein we’ll see that the plugin is responsible for ensuring the state of our infrastructure matches what we’ve specified. In this case, the group plugin will create 5 instances. Let’s take a look at them as well. What directory do you think they’ll reside in?

$ build/infrakit group watch cattle.json
watching cattle

We’ll see some action in the the window watching the infrakit-group-default plugin, which is our instances being created.

INFO[2592] Watching group ‘cattle’
INFO[2603] Adding 5 instances to group to reach desired 5
INFO[2603] Created instance instance-1492454290734052860 with tags map[infrakit.config_sha:006438mMXW8gXeYtUxgf9Zbg94Y= infrakit.group:cattle project:infrakit tier:web]
INFO[2603] Created instance instance-7546878414114322916 with tags map[infrakit.config_sha:006438mMXW8gXeYtUxgf9Zbg94Y= infrakit.group:cattle project:infrakit tier:web]
INFO[2603] Created instance instance-404669573633486624 with tags map[infrakit.config_sha:006438mMXW8gXeYtUxgf9Zbg94Y= infrakit.group:cattle project:infrakit tier:web]
INFO[2603] Created instance instance-4031626780300815760 with tags map[tier:web infrakit.config_sha:006438mMXW8gXeYtUxgf9Zbg94Y= infrakit.group:cattle project:infrakit]
INFO[2603] Created instance instance-6833202584168745085 with tags map[infrakit.config_sha:006438mMXW8gXeYtUxgf9Zbg94Y= infrakit.group:cattle project:infrakit tier:web]

Now we’ll look for those instances ourselves using the CLI. We only need to use the short name of the group with the inspect command

$ build/infrakit group inspect cattle
ID LOGICAL TAGS
instance-1492454290734052860 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-4031626780300815760 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-404669573633486624 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-6833202584168745085 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-7546878414114322916 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web

And lovely, there they are.

Eagle-eyed readers may also notice that you can use the Instance Plugin to return back available instances across all groups.

$ build/infrakit instance — name tutorial-instance-file describe
ID LOGICAL TAGS
instance-1492454290734052860 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-4031626780300815760 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-404669573633486624 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-6833202584168745085 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web
instance-7546878414114322916 — infrakit.config_sha=006438mMXW8gXeYtUxgf9Zbg94Y=,infrakit.group=cattle,project=infrakit,tier=web

Let’s pretend that we’re increasing the size of the cluster and create a second cluster definition file that we’ll use to grow the members.

$ cp cattle.json cattle2.json

Once inside, change the note and the “Size” to a larger number.

$ diff cattle.json cattle2.json
3d3
< “Size”: 5
 — -
> “Size”: 15
11cf10
< “Note”: “Instance properties version 1.0”
 — -
> “Note”: “Instance properties version 1.1”

Using the build in functionality of InfraKit, we can check what the proposed changes are to the infrastructure, similar to what we see in Terraform plan mode.

$ build/infrakit group describe cattle2.json
cattle : Performs a rolling update on 5 instances, then adds 10 instances to increase the group size to 15

If you watch the group plugin window, you’ll see the rolling update take place. I’ve also left a bit of instance trash laying around on purpose in order to observe the scaler cleaning up undesired instances.

INFO[0189] Executing update plan for ‘cattle’: Performs a rolling update on 5 instances, then adds 10 instances to increase the group size to 15
INFO[0199] Scaler has quiesced
INFO[0199] Found 5 undesired instances
INFO[0199] Destroying instance instance-1492454290734052860
INFO[0208] Adding 1 instances to group to reach desired 5
INFO[0208] Created instance instance-5018328134887569524 with tags map[infrakit.config_sha:BejX0kJqx-RnTybBxag_F6QKo88= infrakit.group:cattle project:infrakit tier:web]
INFO[0209] Scaler has quiesced
INFO[0209] Found 4 undesired instances
INFO[0209] Destroying instance instance-4031626780300815760
INFO[0218] Adding 1 instances to group to reach desired 5
INFO[0218] Created instance instance-2689975102812625827 with tags map[infrakit.config_sha:BejX0kJqx-RnTybBxag_F6QKo88= infrakit.group:cattle project:infrakit tier:web]
INFO[0219] Scaler has quiesced
INFO[0219] Found 3 undesired instances
INFO[0219] Destroying instance instance-404669573633486624

It’ll take a minute or two to build your new instances, after which you’ll see ‘update cattle completed’.

InfraKits primitives are designed to enforce the state of your declarative infrastructure, so let’s simulate an instance being removed by deleting it from the filesystem.

$ rm tutorial/instance-1479853841972663246 tutorial/instance-2689975102812625827 tutorial/instance-3020678231588092304

If you check in the tutorials directory, you’ll notice 3 new instances spun up with newer timestamps. That’s because the InfraKit plugins are still watching your definition of the “cattle2” infrastructure, and are polling to ensure that the cluster stays in the declarative state.

$ ls -al tutorial
total 104
drwxr-xr-x 1 5 jesse staff 510 Sep 28 16:40 .
drwxr-xr-x 36 jesse staff 1224 Sep 28 16:39 ..
-rw-r — r — 1 jesse staff 654 Oct 22 16:34 instance-5611776789594408094
-rw-r — r — 1 jesse staff 654 Oct 22 16:34 instance-6284096569187905113
...
-rw-r — r — 1 jesse staff 654 Oct 22 16:40 instance-7938961394285548524 ← new instance

Now that we’ve gotten to the end of the tutorial, we can clean up using the destroy command.

$ build/infrakit group destroy cattle
destroy cattle initiated

Great! In follow up efforts, we’ll start to build real, actual infrastructure using the AWS and Terraform tools that the team has been hard at work building for us over the last few weeks. Looking forward to working on those tools with you!

Click me!

Thanks for reading, and as ever — click the heart below if this helped!