At Bitrise we have multiple organizations registered on GitHub with several open source repositories. Our developers and managers often need to access a quick overview of the current status of repositories and/or issues.
Written by Tamás Srancsik, originally published on the Bitrise blog.
We can process data the way we did with ClickUp, using JSON and HTTP type providers or we can opt for the official GitHub library, Octokit to communicate with the API. With Octokit, once authenticated you can easily build your reports in Ruby, Node.js and .NET languages but there are third party libraries for other frameworks too.
The next few paragraphs will introduce methods for processing data from GitHub API.
Octokit in F#
The .NET documentation encourages us to write code in C# but a lazy coder like me prefers fewer lines. Also, we do not need to compile the code, our beloved F# Interactive will do the job instead.
We need a couple of packages:
- Octokit to get the data
- optionally, Deedle for data frames and statistics
- Xplot to present our reports.
open Octokit
open Deedle
open Xplot.GooglechartsConnect
You can try the API without authentication but that will limit you to 60 request an hour. Authentication is easy once you registered your application at GitHub. GitHub requires a valid User-Agent value, so use the name of your application. Use your token the following way to establish connection.
let appName = "Bitrise-GitHub-App"let client = GitHubClient(new ProductHeaderValue(appName));let token = 'abc123...'
let tokenAuth = Credentials(token)
let client.Credentials = tokenAuth
Get that data
client.Repository.GetAllForOrg() method needs only the name of the organization as parameter. The signature of Task<IReadOnlyList<Repository>>tells us this is a task that is generating a list of repositories. The following lines count the repositories of bitrise-steplib where you can find code of our workflow steps.
let org = "bitrise-steplib"let repos = client.Repository.GetAllForOrg(org)// execute task
let repoResults = repos.Result// number of repositories
repoResults.Count
Process the data
For F# Deedle is what pandas is for Python: it has all the functions to process data in a tabular format. Let’s draw a table of the number of forks, open issues and stargazers to understand which step has the most active community.
repoResults// filter attribute
|> Seq.map (fun r -> (r.FullName, r.ForksCount, r.OpenIssuesCount, r.StargazersCount))// build a data frame from F# record
|> Frame.ofRecords// unfortunately it will not inherit the keys
|> Frame.indexColsWith ["repository"; "forks"; "open issues"; "stargazers"]// descending order
|> Frame.sortRowsBy "stargazers" (fun v -> -v)// Google Charts table
|> Chart.Table
We can easily collect high level figures with Deedle and its Stats namespace. This last contains members for several statistical indicators like mean, standard deviation or skewness etc.
let options = Options(colors = [|"#6B25BC"; "#60D0C4"|])repoResults
|> Seq.map (fun r -> (r.Owner.Login, r.ForksCount, r.OpenIssuesCount))
|> Frame.ofRecords
|> Frame.indexColsWith ["organization"; "forks"; "open issues"]// total numbers by organization
|> Frame.aggregateRowsBy ["organization"] ["forks"; "open issues"] Stats.sum
// we need only the numerical columns for the chart
|> Frame.dropCol "organization"
|> Chart.Column
|> Chart.WithTitle "Total number of forks and open issues on repositories of bitrise-steplib"
|> Chart.WithOptions options
|> Chart.WithLegend true
Stay tuned for the next article on data wizardry titled Understanding Variance in User Attributes coming soon on the Bitrise Blog.
