GroveAI #1: Generating GitHub PR Descriptions as a Team

6 min readFeb 24, 2025

tl;dr Teams should build their own LLM tooling, starting with GitHub PR Description generators.

Quick Preamble

This is the first (of hopefully many to come) blog post covering how our engineering team at grove.city leverages AI-enabled tooling to speed up our workflow. If you want to learn what we do and why we do it, check out one of our other recent posts.

The goal is to keep these short, tactical, actionable and most importantly, opinionated, to how we work. These are not intended to be off-the-shelf solutions. If your team wants to adopt something similar, you’ll likely need to modify it.

The Situation — Writing GitHub Pull Request Descriptions
The Problem — There is no one-size-fits-all solution
The Idea — Make it easier to use our own PR Templates
The Results — Productive developers who save time
The Next Step — Things to keep in mind

The Situation — Writing GitHub Pull Request Descriptions

The software industry is plagued by the zeitgeist of “git commit best practices.” We’ve mostly moved from git messages to GitHub PR Descriptions, but the same idea applies. It’s necessary, it’s useful and there are a lot of opinions as to how it should be done. It’s simple but takes a non-trivial amount of time to do well. There are universal best practices, but they are heavily dictated by team-specific culture. It’s the perfect recipe for bikeshedding (i.e. disaster).

GitHub Pull Request descriptions are a high-level overview of the changes being proposed. In 2025, with a task that involves summarizing or condensing a large amount of data into something much smaller, you should have the instinct or muscle to start with an LLM. If you don’t, it’s time to 🏋

The Problem — There is no one-size-fits-all solution

We tried products like reviewepad when it first came out, and coderabbit.ai when it went mainstream. The first few times you see it, it feels like magic. A few days in, it’s cool. A couple weeks later, it looks cool, but you realize it’s just noise that gets in the way. Eventually, you just gloss over it because it becomes that false positive alert you learn to ignore. One day, you’ve had enough and you just turn it off.

These are the only two products our team has tried for PR description generation, and I’m sure they work well for some, but not for us.

Harper’s recent post captures one of the biggest issues in LLM-enabled workflows really well:

My main complaint about these workflows is that it is largely a solo endeavor — i.e. the interfaces are all single player mode.
[…] These workflows are not easy to use as a team […]
[… ] There is so much opportunity to fix this and make it amazing […]

Everyone is blogging about how Personal Software Is Becoming A Trend, yet at the same time, everyone is trying to build building blocks and tooling for others to reuse. I’d say it’s analogous to preparing a todo list or roadmap: the only one you’ll ever be satisfied with is the one you create for yourself.

The Idea — Make it easier to use our own PR Templates

The two main repositories that Grove contributes to have PR templates. You can find them at poktroll/pr_template.md and path/pr_template.md.

The way these were set up and designed are specific to our workflows, our culture, were iterated over a couple of years, and will also likely change in the future. Since we know what we’re looking for, there was an opportunity to reduce some of the steps without resolving to full automation.

We’re using a tool called diff2html to capture a diff in json format like so:

git --no-pager diff main -- \
    ':!*.pb.go' ':!*.pulsar.go' ':!*.json' ':!*.yaml' ':!*.yml' \
    | diff2html -s side --format json -i stdin -o stdout \
    | pbcopyb

Then, we use a Claude Project with the following instructions

You are a senior developer with good written communication skills.

Your job is to prepare GitHub Pull Request Descriptions.

You will be provided a GitHub PR diff created using the following tool:

```bash
git --no-pager diff main -- \
    ':!*.pb.go' {OTHER IGNORED FILES} \
    | diff2html -s side --format json -i stdin -o stdout \
    | pbcopyb
```

Provide a GitHub PR description of the following format:

```markdown
## Summary
< One line summary >

### Primary Changes:
- < core changes # 1 >
- < core changes # 2 >
- ...

### Secondary changes:
- < secondary changes # 1 >
- < secondary changes # 2 >
- ...
```

Considerations for the output:

- Primary changes should include the main goal(s) of the Pull Request
- Secondary changes include misc changes (e.g. documentation updates, cleanup, etc)
- Escape key terms with backticks
- Keep the bullet points concise
- Limit the number of bullets to 3-5

The Results — Productive developers who save time

I’ve attached a couple of screenshots, but would say that the results speak for themselves 🔥

It’s not perfect, and never will be, but it’s actual value add that creates better PR descriptions for the code reviewer and saves time for the code author.

On a team of 9, with 6 full-time engineers, even if we cumulatively save 1 hour a day, this tiny thing alone pays for all of our AI subscriptions.

The Next Step — Things to keep in mind

There are two key things to keep in mind:

Prompts are living scripts
Don’t automate too early

Prompts are living scripts. They’re never one-and-done. We might need to update the git diff to ignore other files, or there may be an edge case specific to our codebase that the LLM did not tend to correctly. Whenever something like this happens, it’s an opportunity to update the prompt and share it with the team.

Don’t automate too early. Any engineer reading this is probably thinking: “We could have a PR label that triggers a workflow that does this automatically!” You could, but do you need to?

Elon Musk’s fifth rule is to leave automation to the very end:

“[Automate] comes last. The big mistake in [my factories] was that I began by trying to automate every step. We should have waited until all the requirements had been questioned, parts and processes deleted, and the bugs were shaken out.”

Automation always takes more time than expected, requires maintenance, and is harder to adapt if the process is still in flux. We’re already saving tons of time, so let’s just take the win.

In addition, there are smaller steps that can be taken on the path to full automation without all of its downsides.

For example, I personally created a few Makefile helpers for this workflow in the process of writing this blog post. One of our backend engineers, Pascal, created a script and shared it with everyone via this GitHub gist.

This is probably the simplest LLM workflow we use, but it’s the first one we manage to standardize across the entire team.

If you want to follow along for more from myself or the rest of our team, subscribe to our blog: https://medium.com/decentralized-infrastructure

Grove