Reviewabot — Your AI Pull Request Reviewer

Maybe you are in the same situation as me? I have a lot of public and private repositories on GitHub where I’m the only contributor. That’s fine, except there is no one there to review your Pull Requests. But if GitHub Copilot can help you write code, then we should also be able to use the same models to make a generative AI based on an OpenAI GPT that can review your code. So that’s what I made.

Reviewabot is a little tool that uses GPT-4 to review and approve your Pull Requests. It reviews any kind of code and also your README.md and other changes.

Will it replace a good old review by your colleague? No! But I think it’s better than nothing. And nothing is what you get when you are working on your own. So a little AI review shouldn’t harm — it might actually find something your experienced eyes might have missed. 😀

Code Review Example

After you’ve configured your repository to use Reviewabot (See instructions), the flow is just like any other Pull Request Review.

In this example, I’ve created a user named Reviewabot and added it as a contributor to the repository. I’ve then created a dummy PR with changes to 3 files. I would expect some of the changes to be caught in a review.

A PR with 3 filechanges, ready for a review

Let’s ask Reviewabot for a review. That’s done by simply assigning the review to the Reviewabot user.

Request the review to trigger Reviewabot

Sit back, grab a ☕️, and wait for Reviewabot to complete the review. Here’s the result.

The review result

Since we don’t want Reviewabot to block you from merging tour PR, it will always approve your PR, no matter how bad it is. Here you can see that Reviewabot approved your changes and you can merge the PR.

Approved PR

Behind the Scenes

Of course, you can see the code behind Reviewabot. It’s Open Source! Fork it and do whatever you want with it. If you have ideas or just end up using this, then I would like to hear about it. It would make me proud and happy.

https://github.com/simon-k/reviewabot

This is how it actually works. It’s surprisingly simple!

The Reviewabot flow

I start off by getting the PR diff. That’s an HTTP GET in the GitHub REST API. Next, I just send the diff to OpenAI GPT-4 and ask it for a review. I was very surprised to find out that the model understands a git diff text, which is pretty technical. Finally, I submit a PR Review with an Approved event and the review as a comment.

Semantic Kernel is configured with a metaprompt that describes what the model should look for when doing a review. This metaprompt probably needs some refinement along the way, or maybe even be configurable so it suits your preferences. That is on my TODO list.

How to get it

Reviewabot is available on the GitHub Marketplace as an action, so it is really simple to use.

https://github.com/marketplace/actions/reviewabot

There are a few steps you need to go through.

1. Create a new GitHub user that Reviewabot will use to submit the review with. Create a PAT for the user that has permission to read content and update pull requests.
2. Add the new user to your repository as a contributor.
3. Add a secret with your OpenAI API key and the new users PAT.
4. Add the Reviewabot GitHub action from the marketplace.

Here’s an example of a workflow that uses the action. It will review the code when a PR is created. You could also make a workflow that triggers when you assign the PR Review to your review user. See more examples in the action readme.

name: Review Pull Request
on:
  pull_request:
    types: [opened]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
    - name: Run Reviewabot
      uses: Reviewabot/action@v2.0.0
      with:
        open-ai-key: ${{ secrets.OPENAI_KEY }}
        reviewer-pat: ${{ secrets.REVIEWER_GITHUB_PAT }}

That’s it… really, there is not much more to it. And it should work on both Ubuntu and Windows runners.

What’s next?

I have a few ideas for improvements.

First of all, Reviewabot should be able to comment on lines instead of files. That is an obvious improvement. It’s just a little complicated and time-consuming to implement, so I skipped it for now.

I think it would be a nice feature if the user — you — could specify the behavior and guide Reviewabot on what it should look for. Maybe you don’t care about comments in the code. Maybe there is something else that is more important to you. Of course, you should be able to customize it.

And finally, I suspect that the new OpenAI o1 model will be better than GPT4o for doing code reviews. I’ll try it out as soon as I can get my hands on it.

Disclaimer

This tool was made over a weekend in my spare time. Please don’t judge me on any bugs you might find. Instead, please let me know by creating an issue in the GitHub repository or commenting on this article. 🫶

Thank you ❤️

I’ll bet that GitHub is already working on something similar to this. Why wouldn’t they? It’s such an obvious next step to their GitHub Copilot. Reviewabot is limited because it doesn’t have the entire context of the codebase and everything the repository contains, like all the issues and the wiki.

And if you look around, you might be able to find other tools that does a similar or better job than mine. Try to look at GitHub Copilot Extensions. But Reviewabot took a weekend to make and it was a lot of fun.

I hope you got inspired by this. Please let me know what you think in the comments below.

Edit: This article was updated September 26th 2024 with a new description of the Reviewabot action from the GitHub marketplace.

Edit: Since September 29, 2024 Reviewabot also supports Azure OpenAI