Summary of GSoC’21 with Cloud Native Buildpacks
If you are a student trying to begin with open source or already have begun it, Google Summer of Code is one such program that can help you grow into a better contributor, communicator and of course, a better coder. This blog is the summary of what we learned and built in the Google Summer of Code, which will help you and future contributors learn from our experiences.
The joy of contributing doubles when you get to know that programs like Google Summer of Code not only help you learn but also earn while learning. Another added advantage while contributing to open-source is that you get more practical knowledge of almost all professional skills you might need when you leave your college and join as a professional. This spirit of learning by doing and my interest in Cloud Native projects like Kubernetes and Buildpacks made me involve in Google Summary of Code again this year as a Student Developer for Cloud Native Buildpacks.
The Project: Cloud Native Buildpacks
Buildpacks are pluggable tools that can be used to transform the source code to OCI compliant images (Docker images) that can run on any cloud. It is a Cloud Native Computing Foundation incubating project that provides a higher level of abstraction compared to Dockerfiles, decoupling the entire operational overhead and automating the process of rendering the applications production-ready.
This summer, I worked with Joe Kutner (my amazing mentor for GSoC’21) to solve an issue where we didn’t have the mechanism to inform the Buildpack Registry Index repositories if they are updated with the latest Github Actions version or not. Through this project, we aimed to implement Automating Pull Request to Registry Index — whenever a new version of the github-action repo is released by automating staging tests of new action versions.
How it was built?
The project consists of two parts — the first part required clearing of verifying namespaces in buildpacks/github-actions. What we required was actually that the verify Namespace Owner Action should accept a blocklist array of namespaces names to disallow. This would be used when add-if-missing is set to true.
This was implemented as a new field on the config struct in verify_namespace_owner.go, and then validated in the same way as AddIfMissing in that file.
The PR that fixed it is embedded below and after this was merged, we moved to the primary goal of setting up a GitHub Action that automates the updating of the version of buildpacks/github-actions in buildpacks/registry-index.
Let’s move to the second and actually the primary part of the project that we built but before that let’s understand the issue first we are trying to solve.
If we take a look at this PR, we will realise a couple of things — first, that there’s a repo called buildpacks/registry-index whose workflow files depend on buildpacks/github-actions release tags and second, that every time the new version of buildpacks/github-actions is released, we need to manually update the version in the workflow of buildpacks/registry-index too by creating a PR as in this case
However, this manual method was tedious and can also cause breaking issues if we somehow skip updating the version.
This gave birth to the issue embedded below:
In order to fix this, we considered various approaches and below were the two proposed approaches:
Proposed Solution 1:
* In the buildpacks/github-actions repo, add another workflow that triggers `on-release`
* This workflow can be used to create an issue titled somewhat like `Version update` in builpacks/registry-index repo.
* In the `buildpacks/registry-index` repo, we can add either in the existing workflow or a new one to be triggered whenever an issue with the above title is created.
* We can use a tool that I built called sedpy (or sed itself) to in-place edit the workflow file with the latest version, this change can be committed and a PR can be created on its own repo (that is buildpacks/registry-index). Somewhat similar is done by one of the projects that can be found here
* We can either directly push (as done in my above project or create a PR and close the issue)
Proposed Solution 2:
* In the buildpacks/github-actions repo, add another workflow that triggers `on-release`
* Instead of creating an issue, we can try to update the version and create a PR directly from here.
* We can use some references from these clients : (*PullRequestsService) Create and Release Event.
Final Solution
The final solution somehow closely went with the Proposed solution #2 and so what we decided was:
- Adding a workflow in buildpacks/github-actions that will dispatch event on https://github.com/buildpacks/registry-index on every release of buildpacks/github-actions
- this will trigger the workflow in registry-index to create a PR to update the https://github.com/buildpacks/registry-index/blob/main/.github/workflows/register-buildpack.yml
Let’s take a look at how our workflow looked in buildpacks/github-actions first:
name: Creating the PR to update the version of buildpacks/github-actions on buildpacks/registry-index
on:
release:
types: [published]
jobs:
myEvent:
runs-on: ubuntu-latest
steps:
- name: Repository Dispatch
uses: peter-evans/repository-dispatch@v1.1.13
with:
token: ${{ secrets.DISTRIBUTION_GITHUB_TOKEN }}
repository: buildpacks/registry-index
event-type: release-eventIf we look at this workflow, we observe a couple of things:
- It is triggered
on-release - It uses a hook called Repository Dispatch that will work on the trigger of
event-type: `release-event`
Below is the PR that added this workflow file to buildpacks/github-actions
However, the next part of our work was to be done in buildpacks/registry-index where the above hook would act as a trigger for the new workflow that we will add in registry-index which will be used to create the PR that updates the version of buildpacks/github-actions.
Here’s how the workflow file looked when we added in buildpacks/registry-index:
name: Repository Dispatch on release of buildpacks/github-actions
on:
repository_dispatch:
types: [release-event]
jobs:
myEvent:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Update Version of buildpacks/github-actions
run: |
NEW_VERSION=$(curl -s -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/buildpacks/github-actions/releases/latest | jq .name -r)
sed -i 's/compute-metadata:[0-9].[0-9].[0-9]/compute-metadata:'"$NEW_VERSION"'/g' .github/workflows/register-buildpack.yml
sed -i 's/verify-namespace-owner:[0-9].[0-9].[0-9]/verify-namespace-owner:'"$NEW_VERSION"'/g' .github/workflows/register-buildpack.yml
sed -i 's/verify-metadata:[0-9].[0-9].[0-9]/verify-metadata:'"$NEW_VERSION"'/g' .github/workflows/register-buildpack.yml
sed -i 's/add-entry:[0-9].[0-9].[0-9]/add-entry:'"$NEW_VERSION"'/g' .github/workflows/register-buildpack.yml
sed -i 's/yank-entry:[0-9].[0-9].[0-9]/yank-entry:'"$NEW_VERSION"'/g' .github/workflows/register-buildpack.yml
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3.10.0
with:
token: ${{ secrets.DISTRIBUTION_GITHUB_TOKEN }}
commit-message: update version to latest release of buildpacks/github-actions
title: Update Version of buildpacks/github-actions
body: Updates version of buildpacks/github-actions to reflect release changes in workflows of buildpacks/registry-index
branch: update-version
base: mainLet’s now reflect on the logic of this workflow file that does the trick for us!
- We can observe the trigger of this workflow is when it receives the
repository_dispatchof typerelease-event - The next tricky thing to figure out was the logic to find the latest version github-actions and my mentor Joe Kutner really helped me out with this and therefore we went with the below approach:
NEW_VERSION=$(curl -s -H “Accept: application/vnd.github.v3+json” https://api.github.com/repos/buildpacks/github-actions/releases/latest | jq .name -r).This approach basically used curl to find the latest tag name and we store it in the variable NEW_VERSION - Next interesting thing was to find the logic for in-place replacement of the NEW_VERSION with the older version in the https://github.com/buildpacks/registry-index/blob/main/.github/workflows/register-buildpack.yml
- This took a lot of time actually to come out with the right regex patter and sed replacement and my mentor Joe again helped a lot to get an idea of it.
- We can see this logic is implemented in a pattern like this:
sed -i ‘s/compute-metadata:[0–9].[0–9].[0–9]/compute-metadata:’”$NEW_VERSION”’/g’ .github/workflows/register-buildpack.yml - After figuring out all this, the workflow basically just deals with how to create the PR with the above changes which were quite easily done.
With all these changes, we added the above workflow in the form of the below-mentioned PR:
Project: Live-in-Action!
The real challenge for any project that we build comes when it actually goes in action and I am sharing here two PRs that tells the success of how our Project has worked with every release of buildpacks/github-actions since then:
- https://github.com/buildpacks/registry-index/pull/1618
- https://github.com/buildpacks/registry-index/pull/1661
Future Scope and Improvements
- One of the primary improvements we are working on is to make this GitHub Action work for
staging-registry-indexas well. - Another improvement is that we’ll need to write a script that can test the staging-registry-index after a new version is merged. This seems to be simple in principle, but there’s probably a lot of little things that need to be set up. This is another thing that we have already started the work on with my mentor Joe Kutner.
- In future, I plan to continue my contributions to the Buildpacks community and in fact, will try to encourage more students to come and be a part of this community where we have contributions welcome from everyone!
Learnings
- One of the most memorable experiences of my life has been to work with a mentor as helpful and kind as Joe Kutner. He is the Software Architect of Salesforce and a founding member of Buildpacks but working with him has enriched in me such values that are beyond technical nucleus — his eagerness and passion to help and answer my questions has made me more kind and humble to help others in the same way.
- Technically speaking, while the initial journey helped me learn nuances of Golang, the second part was more involved with Regex, scripts and of course making GitHub Actions.
- Apart from that, the bonding and the social skills developed due to community meetings at Buildpacks and the inclusiveness of every member of this community just made me a better open-source student.
Before this blog comes to its end, I would like to express my gratitude to my mentor Joe who has been a constant support in not only answering my every question but also helping and uplifting me whenever I felt stuck. Immensely grateful also to the entire Buildpacks and CNCF community which has helped me push my boundaries and always instilled in me the spirit of giving it back to the community!
And yes, thank you so much to all the readers who stayed this long to read this part of the blog and I am really looking forward to your feedback and suggestions 😃
Also, if you are interested in knowing about my previous year’s GSoC’s journey — where and how I began with it — you may find this blog useful :))
