Announcing Popper 2.0 — Local Github Actions Executioner in Python
TLDR; Popper 2.0 is a workflow execution engine based on Github Actions with extensions that allow users to run actions in other runtimes in addition to Docker.
The goal of the Popper project is to aid users in the implementation of experimentation pipelines that follow a DevOps approach. Last year, we release version 1.0 of the Popper CLI tool, which allows users to specify pipelines in YAML syntax (see example below).
As we kept incorporating user feedback, the YAML-based pipeline definition syntax (shown above) kept evolving and, over time, started to look like a workflow specification format; we decided it was time to embrace workflows properly. Around this time, Github released Github Actions (referred from this point on as GHA), a workflow language and code execution platform. The GHA workflow syntax, being based on HCL, is one of the simplest workflow formats out there (see example below), which makes it a great way for implementing workflows in many distinct domains. In addition, the open actions specification allows anyone to implement distinct functionality, providing a great extensible framework that can address multiple needs.
Instead of creating yet another workflow language, we decided to leverage GHA’s syntax as the format for specifying workflows and make this the basis of Popper 2.0. Thus, Popper 2.0 is a workflow execution engine based on Github actions(GHA). Popper workflows are defined in HCL syntax and behave like GHA workflows. The main difference with respect to GHA workflows is that a Popper workflow can execute actions in other runtimes in addition to Docker. The workflow language is strictly a superset of GHA workflow language so Popper can run a GHA workflow locally as if it was being executed by the GHA platform.
In addition to running a GHA workflow locally, the goal for the Popper 2.x series is to provide the following functionality:
- Runtimes. Actions can execute locally on the host where the
poppercommand runs (i.e. "outside" a container, thus not strictly depending on having Docker installed). We are working in adding support for other runtimes are such as
rkt, Vagrant, Singularity and others (see this project for more).
- Actions catalog. Provide users with a searchable catalog of available actions and workflows so that they can re-use as much as possible what others have already done.
- Continuous integration. Generate configuration files for distinct CI services, allowing users to execute Popper workflows on these services (Travis, Jenkins, Gitlab and Circle supported).
- Scaffolding. Aid in the implementation of new actions and workflows, as well as extending existing ones.
- Action library. As part of our efforts, we maintain a list of actions available at https://github.com/popperized.