[OUTDATED POST] Announcing Popper 2.0 — Local Github Actions Executioner in Python

Ivo Jimenez
Mar 13, 2019 · 2 min read

Update: Support for Github Actions syntax in Popper has been deprecated. Popper now uses its own YAML syntax for specifying workflows. Read more in the official documentation.

The goal of the Popper project is to aid users in the implementation of experimentation pipelines that follow a DevOps approach. Last year, we release version 1.0 of the Popper CLI tool, which allows users to specify pipelines in YAML syntax (see example below).

As we kept incorporating user feedback, the YAML-based pipeline definition syntax (shown above) kept evolving and, over time, started to look like a workflow specification format; we decided it was time to embrace workflows properly. Around this time, Github released Github Actions (referred from this point on as GHA), a workflow language and code execution platform. The GHA workflow syntax, being based on HCL, is one of the simplest workflow formats out there (see example below), which makes it a great way for implementing workflows in many distinct domains. In addition, the open actions specification allows anyone to implement distinct functionality, providing a great extensible framework that can address multiple needs.

Instead of creating yet another workflow language, we decided to leverage GHA’s syntax as the format for specifying workflows and make this the basis of Popper 2.0. Thus, Popper 2.0 is a workflow execution engine based on Github actions(GHA). Popper workflows are defined in HCL syntax and behave like GHA workflows. The main difference with respect to GHA workflows is that a Popper workflow can execute actions in other runtimes in addition to Docker. The workflow language is strictly a superset of GHA workflow language so Popper can run a GHA workflow locally as if it was being executed by the GHA platform.

Image for post
Image for post
Popper 2.0 is runs Github Actions workflows locally on your machine.

In addition to running a GHA workflow locally, the goal for the Popper 2.x series is to provide the following functionality:

  • Runtimes. Actions can execute locally on the host where the popper command runs (i.e. "outside" a container, thus not strictly depending on having Docker installed). We are working in adding support for other runtimes are such asrkt, Vagrant, Singularity and others (see this project for more).
  • Actions catalog. Provide users with a searchable catalog of available actions and workflows so that they can re-use as much as possible what others have already done.
  • Continuous integration. Generate configuration files for distinct CI services, allowing users to execute Popper workflows on these services (Travis, Jenkins, Gitlab and Circle supported).
  • Scaffolding. Aid in the implementation of new actions and workflows, as well as extending existing ones.
  • Action library. As part of our efforts, we maintain a list of actions available at https://github.com/popperized.

For more about Popper, see our Github project or join our Gitter channel.

getpopper

The Popper Blog

Ivo Jimenez

Written by

PhD candidate, @ucsc; @cross_ucsc grad researcher; BSSw Fellow. Working on @getpopper #sciops #devops #openscience

getpopper

getpopper

Topics on practical aspects of reproducibility in computational and data science: SciOps (DevOps in Science), OpenScience, OpenData, and more.

Ivo Jimenez

Written by

PhD candidate, @ucsc; @cross_ucsc grad researcher; BSSw Fellow. Working on @getpopper #sciops #devops #openscience

getpopper

getpopper

Topics on practical aspects of reproducibility in computational and data science: SciOps (DevOps in Science), OpenScience, OpenData, and more.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store