Software Development Metrics Automation Using GitHub Actions
Most Software Engineers would agree on the ineffectiveness of measuring productivity based on lines code, however, I’ ve seen that because of this, the industry has fallen on a cynism thinking that Software Productivity cannot be measured accurately.
Just because something is hard, does not mean it is not possible.
For some of us is quite often the opposite, we thrive on challenges and look to push the boundaries of innovation. As a Software Engineer, I see three main reasons why quantifying Software Productivity would be desirable.
1. For business owners, managers, tech leaders, or engineers looking to evaluate and get feedback on their work and/or project.
2. Open source users, to quickly assess a project before committing to it.
3. Data scientists recognizing that knowledge comes only after a hypothesis can be tested and validated.
Main Software Metrics:
As an introduction to metrics and following a rather arbitrary but pragmatic taxonomy I will classify metrics in 5 categories: Code Lifecycle, Code Quality, Team Dynamics, Project Health & Business Requirements.
The code lifecycle is the simplest of all, and also the less subjective. It rather measures the amount of time your development pipeline takes to implement, as an added bonus it is almost always the easiest to interpret as in most cases the fastest your code lifecycle is the better your project should be. To explain the implementation let’s identify key milestones on the software development lifecycle: first commit (branch creation), pull request created & pull request merged & feature release.
From this milestones is easy, to abstract simple, yet very useful metrics:
- Development time: from the first commit to PR merged.
- Deployment time: from PR merged to release.
- Code review time: from PR created to PR merged.
Now, depending on your CI/CD pipeline additional steps may be added, some examples include unit test coverage threshold on new code (usually between 60% - 90%), E2E and/or integration testing completion based on the feature requirements, code review approvals (highlighting the first review as the most relevant to measure), and user story creation time & assignment.
The two last milestones, create a metric that encompasses the full product development lifecycle.
However, while it is tempting to reduce this time to the most possible. In practice, it should not be, as good development practices would allow a time to plan and mature user stories before actively working on them. As a rule of thumb, the “maturing” would be proportional to the development time.
To finalize the exposition of code-lifecycle metrics, there is another breed o that does not focus on duration but rather on the frequency of the events. The most important of is how often does the team release new code, is it on a daily basis, weekly, monthly, or more?
The rest of the metrics are not as straightforward to implement nor interpret thus, I will only cover them briefly:
Code Quality
Code coverage is fairly well developed, easy to understand and implement, a second step on this is to measure the degree of change in code coverage with new commits, the percentage of new code is well tested and how many “legacy” code is revisited with increased testing.
Using standardized linting, we can get a good approximation on how many tests per lines of code there are, a great idea is to visualize directories and files as a graph with colored nodes based on relative wellness of test coverage.
Code quality does not come from test coverage only but also from collaboration.
The number of comments per PR is a good way to measure that, more advanced use cases include the quality and tone of the comments, assessed using NLP.
Two last comments on code quality: the ratio of bug-fixes and the percentage of high-risk PRs. Here, we start getting subjective but classifying a PR as a bug fix can be as simple as labeling it, more advanced cases include tracing the origin of the bug (git blame) and how many tests were in place before fixing it. As for the high risks PRs they can be determined from the size of the PR, medium risk PRs are big with mostly new code in them or small refactors, high-risk PRs are big reactors.
Team Dynamics
Software is not only about code & engineering but also about people. Sure, a good first step to evaluate the team is to segment the above metrics by contributor but more interesting cases turn around on how the team collaborates with each other.
There are “hidden” productivity metrics that can be measured by individual, to better assess his or her global contribution to the team.
These include: documentation, PR reviews & library or reusable code developed, sure developing a dozen API endpoints is good, but developing an abstraction to catch and monitor errors can be better and more difficult to replace.
Project Health
It can be measured by what direction/services are being worked on using once again a graph, how many new contributors are being added, and how many new issues are created.
Also important is how fast are the issues being resolved (especially, the critical ones), and who is creating them, is it business, QA, developers, or the end-users? Curiously, the answer to that last question depends on the maturity of your project as you would hope that when starting end-users catch and report bugs while on later stages it is your QA and other internal teams.
Business Requirements.
While highly subjective is good to measure to know how many iterations for new features until it becomes stale, how long each user story takes to complete, and how many “pivots” of a new feature occur, this last captures how many change of requirements there are.
Carryover, or what percentage of tasks are passed from one Sprint to the next is also relevant.
Introducing GitHub Actions.
In November 2019, Github released Actions and per its own definition,
it makes it easy to automate all your software workflows, with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.
Github has built a marketplace where you can incorporate to your own projects third-party tools or develop your own. It is with capability that measuring productivity becomes a real possibility. By tracking events (commits, pull requests, code reviews, and more) directly into your repository:
Is possible to automatize, standardize, and report the metrics using code.
Github documentation is amazing, nevertheless let's get to some basic definitions:
Workflow
Is a set of instructions executed by a specific event. This can be as simple as running your unit tests, deploying your infrastructure, or running a particular JS script.
Workflows are composed of jobs: independent set of instructions that can be executed concurrently, each job in turn, is composed of steps that contain one or several commands executed in a Docker container. Normally, Ubuntu flavored.
Triggers
Are the set of events that start workflows besides commits and other usual ones. they are executed by schedule (cron jobs), when issues are created, commented or labeled, or when new tasks are created on projects. At the repository level events such as starring, forking or watching can also create triggers, and there are also trigger from webhooks, manual actions, and much more…
Actions
Actions are developed solutions by top SaaS providers, leading Open Source projects, and motivated individuals that abstract a group of commands in a single request. An example of a really interesting one is Cypress.
A great way to get started to develop your actions are learning labs provided by GitHub in particular the one on Javascript is really easy to follow. Besides that, actions can be implemented in Python, Java & Docker.
Octokit
Is the last piece we need to get started automatizing Software Metrics, it is the official Github API client, and with it, the breadth of possibilities is intimidating. As a rule of thumb if you can do it in GitHub you can do it with Ocktokit.
Implementing a Software Metrics Collector MVP
Designing an MVP is all about simplification
And thus code lifecycle metrics are the natural candidate for implementation. As for reporting them it gets more complicated, my initial thought was to create a simple React dashboard, however, on a closer look, that is more like a full product. Then I thought of a Slack bot, it is handy, quick, and simple to develop. Fortunately, there is one more straightforward approach to it:
Commenting put the metrics directly to GitHub:
My first attempt was commenting to an issue, I thought about creating a particular label opening the issue and closing it once all the metrics were collected.
In the end, things got simpler as I commented directly to the PR.
Dependencies: Besides Octokit to query and interact with GitHub, there was the need to preserve events. I choose Mongo for that and with MomentJS to work with dates I was ready to get started.
Development
First step detecting relevant events:
- Branch creation
- Commits
- Pull requests creation
- Commits during PR
- Merge
First, configure the workflow to listen to those particular events:
# Workflow triggers
on:
push:
branches-ignore:
— master
pull_request:
types: [opened, closed]What is it doing here? We listen to every push except the ones done to master. With this, we handle events 1, 2 & 4.
Why is it ignoring master? it is easier to comment on the issue if is handle through a pull request event.
The second triggers listen to pull requests being opened and closed (yes merging is actually a closure). If we were not to ignore the push to master, we would be creating two pipelines from the merge event.
That handles triggering the pipeline, now let’s differentiate the events:
const github = require(“@actions/github”);
const core = require(“@actions/core”);async function run() { const action = github.context.payload.action; const isNewBranch = github.context.payload.created;
const isOpened = action === ‘opened’; const isClosed = action === ‘closed’;
const pull_request = github.context.payload.pull_request;
const isMerged = isClosed && pull_request.merged; ...
}
That is a JS script that gets all the information required, most is included in the GitHub context, and through very similar code we can get the branch name to identify to what feature it relates.
Now only two more things to go: initialize the DB and comment on the issue:
steps:
— uses: actions/checkout@v2
— name: collect-metrics
uses: ./.github/actions/metrics
with:
repo-token: ${{secrets.GITHUB_TOKEN}}
mongo-password: ${{secrets.MONGO_PWD}}The actions/checkout@v2 is usually the first step of every pipeline, with that we pull the code from GitHub to the Docker container. The second part invokes the metrics action with two “secret” values: the GITHUB_TOKEN is automatically created by the pipeline, and the MONGO_PWD is defined on the repository. It will be used to connect to the DB in a safe way.
Let’s see how to comment on the PR:
const github = require(“@actions/github”);
const core = require(“@actions/core”);async function run() {
… const token = core.getInput(“repo-token”); # Get the token passed.
const octokit = github.getOctokit(token); # Initiate octokit. # Get data relevant to find the PR issue.
const repository = github.context.repo.repo;
const owner = github.context.repo.owner;
const issue = github.context.payload.number # Comment on the issue.
await octokit.issues.createComment({
repo: repository,
owner: owner,
issue_number: issue,
body: body # This data is retrieved from MongoDB.
});
}
As simple as that! Initialize Octokit, find the issue and create a comment :)
Next Steps
Natural first steps would be to build a front-end and export the action to the market place. However, there are a couple of concerns I have in doing that:
1. How to protect the DB, against abusive behavior. This example works because the secret is stored inside my project, however, how to safely open a connection from other projects?
2. Not many projects would gladly share their GitHub tokens, even more important, there might some restrictions from GitHub on doing that.
3. Marketing a new action is though, how to know what front-end to build if the action has no users
Based on that, I decided the best way to go is the option to evaluate Open Source projects’ health instead.
I imagine a simple search Google-like UI connected to Github's API. Where a user can select a repository and get some metrics on its health. Since most of the methods to the API would be open, I might be able to get some valuable data from them.
Maybe at the bottom, I could split the section with “featured” repos based on recent activity and possibly a “ranking” system in the second section.
Advertisement could be a good source of side revenue while, while the path of integrating it natively to GitHub would be less direct. For projects, a good first step would be to embed the metrics in the ReadMe via a snippet.
While for contributors I’m thinking a more ambitious approach:
How about getting recommendations on what Open Source to collaborate based on your commit history?
Thank you for reading.
