What’s New in Vagga 0.7.0

6 min readJan 19, 2017

We are excited to announce the next version of vagga, the containerization tool that makes setting up development environments easy and fun.

This release contains more than 250 commits from 27 contributors. Following are some highlights.

Persistent Volumes

Sometimes you want to store some data in a database run by vagga or you want file uploads to be kept somewhere aside of /tmp directory. Since vagga isolates containers from your host system there were only two options:

Configure your database or storage to some subfolder inside the /work
Use !BindRW /work/dir volume to alias some folder inside the container to a more conventional for the database

Now you have a third option: !Persistent volume. To make it work you need to declare a volume:

containers:
  ubuntu:
    setup:
    - !Ubuntu xenial
    # ...
    volumes:
      /var/lib/sql-data: !Persistent sql-data

When a command in the container is run for the first time the volume is created. It doesn’t get rebuild even if your container changes.

Technically a volume is created in .vagga/.volumes/volume-name or in the storage-dir if you had configured one. But the idea is that don’t need to know that. If you need to clean volumes, you might want to run clean command:

vagga _clean --unused-volumes

The command above cleans all volumes that are no longer defined in vagga.yaml. And you can remove all the volumes and get them rebuilt on the next run:

vagga _clean --volumes

See documentation for more details, see postgres example to get a feeling of how all these options might get used together.

Network Isolation

We have some network testing facilities in vagga since version 0.2. And they already has influenced some of our decisions. But the interface is a little bit complex and poorly documented yet (contributions are welcome!).

This release brings simple to use isolate-network options to vagga command line and command definition. For example, if you want to check if a command run by vagga can work without accessing to the internet you can run:

vagga --isolate-network command-name

In this mode, vagga isolates network in addition to traditional filesystem isolation, providing only a loopback interface for the program. The loopback interface (localhost) provided is a new one, separate from your host system.

If you want tests to always run in this mode, set the flag in the command:

commands:
  test: !Command
    container: some-containerwins
    run: #...
    isolate-network: true

Note: each command invocation gets it’s own loopback interface, meaning that if two test commands run in parallel they don’t interfere each other.

Command-line Options

Previously, vagga had support of pass-thru of options to normal commands, but not to !Supervise kind of commands. This was a problem for two cases:

If what you’re running is a script (and parsing command-line in bash is very hard!)
If you’re running multiple processes (i.e. a !Supervise)

We evaluated multiple different approach to the issue and the one which is apparently superior to others is using docopt to define the options and environment variables to pass them around. Let’s take a look how to define an option:

commands:
  test: !Supervise
    options: |
      Usage: vagga test [--redis-port=<n>] [options] [<tests>...]

      Options:
        -R, --redis-port <n>  Port to run redis on [default: 6379]
        <tests> ...           Name of the tests to run. By default
                              all tests are run
    children:
      redis: !Command
        container: redis
        run: |
          redis-server --daemonize no --port "$VAGGAOPT_REDIS_PORT"
      first-line: !Command
        container: busybox
        run: |
          py.test --redis "$VAGGAOPT_REDIS_PORT" $VAGGAOPT_TESTS

The snippet above defines a test suite running two processes redis and py.test. Both are run simultaneously.

When --redis-port=1234 is passed to the vagga test it overrides the port is used for running redis. Other options are passed to the py.test command. As you can see you have a fine-grained control over which options and in which form are passed to each command.

This works for normal commands too. And you’re not required to write a snippet of a shell script in vagga.yaml neither you need to use bash at all, you can use environment variable in whichever way you find desirable. See documentation for more info.

Note: you might notice that this looks like --redis-port options is here to avoid conflicts with other redis instances running on machine. And yes you’re right, this might have been achieved using isolated-network. But there are few cases where the latter cannot work, for example if process needs to access external network for some reason, or if you want to connect to redis from the host system for debugging.

Include Directives

It’s been a long time since vagga first appeared in June 2014. We wanted to put together all those little containers and commands scattered over Dockerfiles, Makefiles, bash scripts and so on. But vagga’s scope has been changed very much over these amazing two and a half years. We still want small and self-contained vagga.yaml for smaller projects, but the biggest project I know about contains over 40 commands in 27 containers. It contains total of more than 1600 lines of yaml (and that doesn’t count few snippets of shell script we factored out from it earlier).accessible in github

So it’s time to make vagga.yaml composable from smaller pieces. In this release we’ve prepared only one tool, it is !*Include tag. It allows to include another yaml in some place of the configuration file. For example:

# vagga.yaml
containers:
  ubuntu: !*Include "vagga/ubuntu.container.yaml"# vagga/ubuntu.container.yaml
setup:
- !Ubuntu xenial
- !Py3Install [pysay]

It includes YAML “semantically” rather than “textually”. Which means if !Include tag is used in a list or a key it only replaces that key and can’t break your file structure or influence sibling keys.

This is just the first step, but with the power of YAML it can already be useful for a lot of things. Let’s study examples of just few of them.

Putting containers and commands in just two files:

containers: !*Include "containers.yaml"
commands: !*Include "commands.yaml"

Boring, right? Let’s put each one to another file:

containers:
  python: !*Include "vagga/python.container.yaml"
  doc: !*Include "vagga/doc.container.yaml"
commands:
  run: !*Include "vagga/run.command.yaml"
  doc: !*Include "vagga/doc.command.yaml"

Still boring. But we’re getting to interesting. Let’s define essential containers here, but put containers and commands running example application elsewhere:

containers:
  python: 
    setup:
    - !Ubuntu xenial
    # ...
  <<: !*Include "vagga/example-containers.yaml"
commands:
  run: # ...
  <<: !*Include "vagga/example-commands.yaml"

If you’re not familiar with YAML enough, you might think that this <<: is a weird smile. But it is a merge key defined in YAML standard (unlike our !*Unpack and !*Include exensions). You can merge multiple things too:

containers:
  <<: [!*Include "norm-cnts.yaml", !*Include "example-cnts.yaml"]
commands:
  <<:
  - !*Include "essential-commands.yaml"
  - !*Include "test-commands.yaml"
  - !*Include "example-commands.yaml"

And surely you can include more things in the included files.

I’m going to stop right here, to give you a chance for some experimentation, just make a note that vagga and vagga _list and
vagga _list --containers list all of the commands and containers respectively. Use it, if you’re in doubt.

Smaller Changes

We’ve done a lot of improvements for running vagga on CI systems. In particular it’s now safer to run multiple instances of vagga in the same directory. By default, vagga stopped to hash permission bits that depend on umask (which often different on CI and on workstations). And more.
!Copy command now accepts include-regex which simplifies copying subset of files. !Depends had grown all the options that !Copy has. And we plan to improve both of them even more in future.
!Py3Requirement now supports -r in requirements, so you don’t have to add aditional !Depends for the included files
!Repo and!AlpineRepo commands help to use custom Alpine Linux repositories (first one works for ubuntu too)
data-dirs option allows to strip unneeded stuff from the container, effectively converting it to a “data container”

And More

Full release notes are accessible in github and upgrading guide is in documentation.

Contributors

We have 27 contributors to vagga 0.7.0. Thank you!

Alexander Koval
Alexey Popravka
Alexey Shmalko
Alex Kuzmenko
Andrea Ferretti
Andy Gayton
Boryslav Larin
Brad Fritz
Florian Gilcher
Gabriele Lanaro
Hans-Peter Oeri
Joe Hillenbrand
Leonardo Cecchi
Lilian A. Moraru
Livio Ribeiro
Luda Lukashevskaya
Łukasz Niemier
Marek Kowalcze
Maxym Vlasov
Parham Doustdar
Paul Colomiets
Prayag Verma
Roman Rader
rootzer0
Simon Bourne
Srinivas Reddy Thatiparthy
Sviatoslav Sydorenko

More features to come. Stay tuned!