Reusable Github Action Workflows II: Integration Testing
In our previous article, we explained how to implement reusable GitHub action and workflow templates. We used Databricks job deployment using dbx as our example task.
Reusable workflow templates are great, and easy to implement, but not so easy to test. Or at least, it wasn’t easy until this article. What it’s done often, is to execute the reusable workflows or actions within the same repository to see if they are implemented right. But is this really testing? Very likely, those workflow templates will be called from another repository. Therefore, what needs to be tested is if they are running as expected when they are called with the right inputs from outside of the template repository.
In this article, we’ll show you how to properly test your workflow template.
Reusable workflows repository → marvelous-workflows
├── README.md <- The top-level README
├── .github
│ └── workflows <- Reusable workflows folder
│ ├── databricks_job_dbx.yml <- Workflow for dbx deployment
│ ├── databricks_job_dbx.md <- Workflow documentation
│ ├── check_dbr_job_run.yml <- Workflow to check databricks job run
│ ├── check_dbr_job_run.md <- Workflow documentation
│ └── dispatch_testing.yml <- Workflow to trigger test run on marvelous-workflows-testing repo
│
│ └── actions <- Reusable composite actions
│ ├── deploy_dbx <- Action for dbx deployment
│ └── setup_env_vars <- Action for setting up env vars
│
└── ml-toolkit
└── check_dbr_job_run.py <- Python script to check databricks job run statusTesting repository → marvelous-workflows-testing
├── README.md <- The top-level README
├── .github
│ └── workflows <- Testing workflows folder
│ ├── test_dbx_reusable_wf_dev.yml <- Workflow to test dev branch
│ └── test_dbx_reusable_wf_master.yml <- Workflow to test master branch
│
├── conf
│ └── dbx_deploy.yml.j2 <- Dbx conf file for test dbr job
│
└── scripts
└── test_dbx_job.py <- Python script for test dbr job
We’ll explain the important pieces below:
- Testing repository → marvelous-workflows-testing
- Test pipelines to run reusable workflows within the actual repository → test_dbx_reusable_wf_master.yml and test_dbx_reusable_wf_dev.yml
- Necessary files for the test pipeline to run → conf/dbx_deploy.yml.j2, test_dbx_job.py
- Extra scripts and workflows required for testing in marvelous-workflows
- A workflow to dispatch test pipeline in the testing repo marvelous-workflows-testing
1. Testing repository
We need a separate repository that will have testing workflows. These workflows simply call reusable actions from the actual repository.
In our example repository marvelous-workflow, we initially have the following reusable workflow: databricks_job_dbx.yml
We write a test pipeline to execute databricks_job_dbx.yml in our testing repository and create a trigger.
2. Test pipeline to run reusable workflows
In our test repository, we have 2 main pipelines:
test_dbx_reusable_wf_master.yml
test_dbx_reusable_wf_dev.yml
These are GitHub action workflows that use reusable workflow from our actual repository marverlous-workflows.
Below, we call databricks_job_dbx.yml workflow from the marvelous-workflows repository. The branch needs to be specified, master, develop etc.
test_dbx_job_deploy:
uses: marvelousmlops/marvelous-workflows/.github/workflows/databricks_job_dbx.yml@develop
with:
deployment-file: "conf/dbx_deployment.yml.j2"
toolkit-ref: "feature/dbx-workflow"
run-job-now: "True"
secrets: inheritSince we can not parameterize the reference branch, we created 2 different test workflows, one for the master branch, and one for develop branch. Ideally, those 2 branches are the most important to test.
3. Necessary files for the test pipeline to run
We are testing a reusable workflow that creates and runs a Databricks job. Therefore we need some extra files to be able to execute the workflow.
- conf/dbx_deploy.yml.j2 → This is the Databricks job config file that needs to be provided as an input to the databricks_job_dbx.yml
- test_dbx_job.py → This is a python script that will be executed in Databricks job.
4. Extra scripts and workflows required for testing in marvelous-workflows
By calling databricks_job_dbx.yml and passing necessary inputs, we are creating and running a Databricks job. We also need to know if the task is successfully completed. To get the status of Databricks run, we added an extra Python script and a reusable workflow to our main repository marvelous-workflow.
The following script takes run_id as input and checks the status. If the status is completed and failed, it gives an error.
marvelous-workflows/ml_toolkit/check_dbr_job_run.py
import time
import requests
import argparse
import os
def databricks_get_api(endpoint, json: dict = None):
domain = os.getenv("DATABRICKS_HOST")
token = os.getenv("DATABRICKS_TOKEN")
print(f"{domain}/api/{endpoint}")
if json:
response = requests.get(
f"{domain}/api/{endpoint}",
headers={"Authorization": f"Bearer {token}"},
json=json,
)
else:
response = requests.get(
f"{domain}/api/{endpoint}",
headers={"Authorization": f"Bearer {token}"},
)
return response
def get_run_status(run_id):
"""Retrives run status for a given Databricks run
Args:
run_id (str): Unique identifier for a specific jon rub
Returns:
str: The status of the job
"""
response = databricks_get_api(endpoint=f"2.0/jobs/runs/get?run_id={run_id}")
if response.status_code == 200:
return response.json()["state"]
else:
print(response.json()["message"])
def get_arguments():
parser = argparse.ArgumentParser(
description="databricks job json file parser and adjustment"
)
parser.add_argument(
"--run_id", metavar="run_id", type=str, default="", help="Databricks run id"
)
args = parser.parse_args()
return args.run_id
if __name__ == "__main__":
for env_var in ["DATABRICKS_HOST", "DATABRICKS_TOKEN"]:
if env_var not in os.environ:
raise EnvironmentError(
f"Must provide value for {env_var} environment variable"
)
run_id = get_arguments()
state_dict = get_run_status(run_id=run_id)
sleep_time = "60"
while (
state_dict["life_cycle_state"] == "RUNNING"
or state_dict["life_cycle_state"] == "PENDING"
):
print(
f"Databricks job status is {state_dict['life_cycle_state']}, sleeping %s seconds"
% str(sleep_time)
)
time.sleep(sleep_time)
state_dict = get_run_status(run_id=run_id)
else:
if state_dict["life_cycle_state"] == "INTERNAL_ERROR":
RuntimeError("Given databricks run failed.")
if (
state_dict["life_cycle_state"] == "TERMINATED"
and state_dict["result_state"] == "SUCCESS"
):
print(state_dict["result_state"])The following workflow takes Databricks run_id and the branch reference. It runs the Python script check_dbr_job_run.py to check the status of Databricks run.
marvelous-workflows/.github/workflows/check_dbr_job_run.yml
name: "Check Databricks job run status"
on:
workflow_call:
inputs:
run_id:
description: Databricks run id
required: true
type: string
toolkit_ref:
description: marvelous-workflows version to use
required: true
default: "master"
type: string
jobs:
check_dbr_job_run:
runs-on: ubuntu-20.04
steps:
- name: Echo inputs
id: echo_inputs
run: |
echo "Branch that was used to start the workflow: ${{ github.ref_name }}"
echo "Github workflow started running: $(date +'%Y-%m-%d %H:%M:%S')"
echo "Run id: ${{ inputs.run_id }}"
- name: Generate token
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
- name: Setup git token
id: setup_git_token
shell: bash
run: |
echo "GIT_TOKEN=${{ steps.generate_token.outputs.token }}" >> $GITHUB_ENV
- name: Get composite run steps repository
id: checkout_marvelous_workflows
uses: actions/checkout@v3
with:
repository: marvelousmlops/marvelous-workflows
ref: ${{ inputs.toolkit-ref }}
token: ${{ steps.generate_token.outputs.token }}
path: actions
- name: Setup env vars
id: setup_env_vars
shell: bash
run: |
echo "DATABRICKS_TOKEN=${{ secrets.DATABRICKS_TOKEN }}" >> $GITHUB_ENV
echo "DATABRICKS_HOST=${{ secrets.DATABRICKS_HOST }}" >> $GITHUB_ENV
- name: install workflow python requirements
run: |
pip install git+https://x-access-token:${{ env.GIT_TOKEN }}@github.com/marvelousmlops/marvelous-workflows@${{ inputs.toolkit_ref }}
pip install wheel
shell: bash
- name: check status
run: |
echo ${{ inputs.run_id }}
mkdir -p model
python -m ml_toolkit.check_dbr_job_run \
--run_id ${{ inputs.run_id }} \
shell: bashIntermezzo
In the marvelous-workflow repo, the action deploy_dbx/action.yml and the workflow databricks_job_dbx.yml are updated in order to give job_id and run_id as workflow outputs. So that we can catch run_id to be used in the workflow check_dbr_job_run.yml.
5. A workflow to dispatch test pipeline in marvelous-workflow repo
The last and very important step is to create a trigger for test pipelines. The triggering mechanism for test pipelines can vary based on the development workflow or project requirements. We implemented a basic git flow. When there is a push to develop or master branches, the test pipeline gets triggered. It can be extended to “PR to develop, PR to master” etc.
marvelous_workflows/.github/workflows/dispatch_testing.yml
name: "Dispatch testing"
on:
push:
branches:
- master
- develop
workflow_dispatch:
inputs:
branch:
required: false
description: Branch of marvelous-mlops to use
type: string
default: "master"
jobs:
test_workflow_push_dev:
if: ${{ github.event_name == 'push' && github.ref_name == 'develop' }}
runs-on: ubuntu-20.04
steps:
- name: Echo inputs
id: echo_inputs
run: |
echo "Branch that was used to start the workflow: ${{ github.ref_name }}"
- name: Generate token
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
- name: Repository Dispatch
id: return_dispatch
uses: codex-/return-dispatch@v1.7.0
with:
token: ${{ steps.generate_token.outputs.token }}
ref: refs/heads/master # branch of marvelous-workflows-testing to use
repo: marvelous-workflows-testing
owner: marvelousmlops
workflow: test_dbx_reusable_wf_dev.yml
workflow_timeout_seconds: 300
- name: Await Run ID ${{ steps.return_dispatch.outputs.run_id }}
uses: Codex-/await-remote-run@v1.0.0
with:
token: ${{ steps.generate_token.outputs.token }}
repo: marvelous-workflows-testing
owner: marvelousmlops
run_id: ${{ steps.return_dispatch.outputs.run_id }}
run_timeout_seconds: 6000
poll_interval_ms: 200
test_workflow_push_master:
if: ${{ github.event_name == 'push' && github.ref_name == 'master' }}
runs-on: ubuntu-20.04
steps:
- name: Echo inputs
id: echo_inputs
run: |
echo "Branch that was used to start the workflow: ${{ github.ref_name }}"
- name: Generate token
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.TOOLKIT_TESTING_APP_ID }}
private_key: ${{ secrets.TOOLKIT_TESTING_APP_PRIVATE_KEY }}
- name: Repository Dispatch
id: return_dispatch
uses: codex-/return-dispatch@v1.7.0
with:
token: ${{ steps.generate_token.outputs.token }}
ref: refs/heads/master # branch of marvelous-workflows-testing to use
repo: marvelous-workflows-testing
owner: marvelousmlops
workflow: test_dbx_reusable_wf_master.yml
workflow_timeout_seconds: 300
- name: Await Run ID ${{ steps.return_dispatch.outputs.run_id }}
uses: Codex-/await-remote-run@v1.0.0
with:
token: ${{ steps.generate_token.outputs.token }}
repo: marvelous-workflows-testing
owner: marvelousmlops
run_id: ${{ steps.return_dispatch.outputs.run_id }}
run_timeout_seconds: 6000
poll_interval_ms: 200
Important!!!
dispatch_testing.yml uses the GitHub app token to trigger the marvelous-workflows-testing workflow from the specified branch.
- Distinct_id (passed automatically) is needed to be able to find the GitHub workflow run id.
After the workflow is triggered, it waits until its status is successful. dispatch_testing workflow is only successful if the workflow it triggers is also successful.
The workflow that gets triggered must include a step that has a distinct id in its name! See marvelous-workflows-testing/.github/workflows/test_dbx_reusable_wf_dev
name: Test Reusable Workflow (ref:dev)
on:
workflow_dispatch:
inputs:
distinct_id:
description: "Unique identifier for the run"If you have an action in a repository that dispatches an action on a foreign repository currently with Github API there is no way to know what the foreign-run you’ve just dispatched is. Identifying this can be cumbersome and tricky.
The consequence of not being provided with something to identify the run is that you cannot easily wait for this run or poll the run for its completion status (success, failure, etc).
For more details, see https://github.com/Codex-/return-dispatch.
