ValueStream: Introducing GitHub Pull Request Metric Reports
ValueStream now supports generating pull request reports for GitHub Repositories. Many of the leading code review metric SAAS’ charge hundreds or thousands of dollars a month for similar data. We believe it shouldn’t cost an arm and a leg to gather and aggregate YOUR data. The ValueStream report generation is able to provide a high level view into pull request performance.
ValueStream ships with CLI tools for pulling and performing aggregations to generate reports. This post uses the ImpactInsights/valuestream pull request information as an example.
First step requires pulling the “raw” pull request data from GitHub. Private repos require generating a Personal Access Token, and setting the VS_PERF_REPORT_GITHUB_ACCESS_TOKEN environmental variable:
$ go run cmd/vsperformancereport/main.go github -org=ImpactInsights -repo=valuestream -out=/tmp/vs-prs.csvINFO[0000] PullRequests.List last=unknown page=1INFO[0000] PullRequests.Get curr=0 last=10INFO[0001] PullRequests.Get curr=1 last=10INFO[0001] PullRequests.Get curr=2 last=10INFO[0002] PullRequests.Get curr=3 last=10...
After the raw data is pulled, the next step is to aggregate it and generate the key pull request performance metrics:
$ go run cmd/vsperformancereport/main.go agg -in=/tmp/vs-prs.csv pull-requestKey,Interval,Owner,Repo,TotalPullRequests,NumMerged,MergeRatio,AvgTotalLinesChanged,AvgDuration,P95Duration,AvgDurationLine,AvgDurationComment2020|3_ImpactInsights|valuestream,2020|3,ImpactInsights,valuestream,1,1,1,428,98687,0,0.0043369440757141265,02020|1_ImpactInsights|valuestream,2020|1,ImpactInsights,valuestream,3,2,0.67,84.66666666666667,3675.3333333333335,0,0.049723958097649984,0.000225281741098385462019|50_ImpactInsights|valuestream,2019|50,ImpactInsights,valuestream,1,1,1,665,8565,0,0.07764156450671336,0.00023350846468184472019|49_ImpactInsights|valuestream,2019|49,ImpactInsights,valuestream,2,2,1,220,251010,0,0.5457733277063436,0.0045464507399914142019|47_ImpactInsights|valuestream,2019|47,ImpactInsights,valuestream,3,3,1,2267.3333333333335,57137.666666666664,0,20.19594165433617,0
By default ValueStream aggregates data by week.
This data can be easily loaded into a spreadsheet and graphed (shown using Google Sheets):
These metrics were chosen as the foundation of pull request observability, and are based on Google’s Four Golden Signals:
- Rate — Answers: “How often are pull requests happening?”
- Duration — Answers: “How long pull requests are taking?”
- Merge ratio — (error ratio) Answers: “What percentage of pull requests are actually being incorporated into the main branch?”
- # Of in progress pull requests (queue depth) — Answers: “How many pull requests are in progress (outstanding)?”. This metric can be derived from the rate and duration using Little’s Law.
Rate
Rate informs how frequently pull requests are occurring. This is essential in order to understand the total cost of pull requests to the organization. ValueStream has a consistent rate of pull requests at ~2 a week.
Duration
Duration determines how long pull requests are taking from Open -> Close. Since pull requests are usually performed as a synchronous step (blocking) within the context of software development life cycle this number is an excellent indicator of the cost of pull requests. A trend up in the average pull request duration is a strong indicator of code or organizational issues.
Merge Ratio
Merge ratio is calculated as the # of pull requests that were merged / the total number of pull requests. This number should be stable or trending up. Pull requests that aren’t merged suggests that code was designed, completely but wasn’t actually incorporated into the main branch. This indicates that resources were wasted on code that never actually made it into the hands of end users.
Pull Request Size
Pull request size helps to understand the relative size of pull requests based on how many lines were changed. Smaller pull requests are easier to review, provide less cognitive overhead, and can be easier to integrate and rollback. This is a pretty weak indicator by itself, but can be an important correlation metric when debugging an increase in pull request durations.
Normalized Durations
Normalized durations helps to understand the time spent in pull requests by line or comment:
These metrics help to compare pull requests of different sizes for a change in performance. These lines should be flat or trending downwards. A trend up suggests that pull requests are taking increasingly longer.
Consider a pull requests of 100 lines with 2 comments that takes 10 hours to review. Duration should *ideally* be linear: A pull request of 1000 lines and 20 comments shouldn’t take more than 100 hours to review. If duration increases it’s a strong signal of code complexity or organizational issues with pull requests or collaborator availability.
Derived Metrics
Some really important metrics that weren’t covered above:
- # Of Pull Requests in Progress — This helps to indicate pull request saturation, when in progress pull requests are increasing and pull request durations are also increasing, it suggests a saturation in reviewers, and more reviewers need to be trained or allowed to review.
- Pull Requests / Issue Ratio — This is really important to understand how many pull requests each issue is incurring.
- Pull Requests / Deployment — Pull Request / Deployment Ratio is a strong indicator of #DevOps maturity. A 1:1 ratio indicates an org is technically capable of deploying changes safely. A higher ratio (N:1) means work is being batched into deploys, suggesting deployment friction, or risk aversion.
Conclusion
Getting highly actionable pull requests metrics should be easy and affordable. GitHub provides these metrics to its users for free. Pull request metrics do not need to be collected in real time to be actionable or useful. This is why we added support to ValueStream to collect these metrics through its new performance report CLI.
We would love to hear your feedback, or feature requests!
Thank you
