A month with GitHub Copilot
Introduction
GitHub Copilot is an AI-powered code assistant that provides suggestions and auto-completion for code as developers write, powered by OpenAI’s Codex (built upon GPT-3). Version 1 has been running a limited trial over the past month to evaluate the effectiveness of the tool. A range of developers (from associate to senior) have been given access to use it for project work.
We’ve seen varied results so far but I’ve grown to love and embrace the tool, discovering its potential as well as the areas where it's really lacking. In this post, I’ll take you through some of the use cases I’ve found and share my findings.
Personal Achievements with Copilot
All of us on the trial have been primarily trying to use it with project work. It hasn’t done well with direct project work as our stack is very bespoke / in-house developed so it’s likely that we’re not in its training data and Copilot will need to learn everything from the context (and this is something we know is limited right now).
Where it has shined, however, is in creating greenfield tools and value-add exercises. There are many things developers have to do manually that could be automated (installing dependencies when onboarding, daily set-up of environments, or just navigating around multiple repos). In an ideal world we would automate as much as possible to let developers spend time on the things that matter but we often just don’t have time to create these tools and it's a hard business sell when managers hear that creating such tooling will take time away from developing new features for their app.
I’m Made by Dyslexia and it really bothers me having to do the same manual tasks every day (even if they only take 15 minutes at the start of the day — that’s 15 mins x 5 days x every dev in the team — a lot of wasted time over the lifetime of a project). Most devs accept that this is the way things are and get on with their day but I believe things can and should always be improved. One task in particular that really irks me is having to switch between hundreds of files at once in our Express/Nunjucks frontend (ie, Controllers, Services, views, locale files etc…). I challenged myself to write a VSCode extension with copilot to make this easier by identifying related files across the project and collecting them through a context menu.
I’ve never created a VSCode extension before and the barrier to entry (that is, researching how to get started and interact with the IDE’s API) would have been too high to do in parallel with project work. Copilot made this process extremely simple. After a couple of prompts and much autocompletion, I had a working extension (complete with tests) within the *same day of starting*. I rely on this extension daily (along with the other frontend devs in my team) and while its hard to quantify how much time it saves, it certainly reduces the mental load of having to search manually.
I’ve also created other scripts with copilot since (all in languages/frameworks that I am not familiar with). This alone is worth the $10 a month it costs, and it’s only going to get better. Improving developer experience is invaluable and removing the common fiddly tasks goes a long way.
What it’s good for
Copilot is mostly marketed as saving developer time by handling boring boilerplate code and tests. It is great at this, but as explained above I would also posit that it's great at smoothing over that uncomfortable transition into a new language or framework. This personally is a great source of my own Imposter Syndrome. That feeling when tasked with developing a new feature in an unfamiliar framework, you may know how to do it with Angular, but this is React and you’re questioning your whole career up to this point. Prompting Copilot with “I can do X to achieve Y in Angular, how do I do this in React?” often gets the job done.
It is also great at writing tests, or completing simple logic as advertised. I hate writing tests (shocker!), and quite often just don’t do it or skimp in personal projects, and it’s the worst part of writing professional code. Of course, tests are valuable and you’re always thankful for them when making changes further down the line. Copilot has removed this mental hurdle for me and I no longer dread writing tests as much (I’ll talk more about this later because there are some important caveats!).
TLDR:
+ Smarter intelllisense/autocomplete
+ Boring boilerplate
+ The micro details of an implementation
+ Translating code between languages/frameworks
+ Tests!
Limitations
Just like many have found with ChatGPT, it can be confidently wrong. It’s very important to question absolutely everything it spits out. I’ve encountered a funny/frustrating situation many times where you can tell it that the code it just hallucinated doesn’t work, it’ll apologize and acknowledge that it doesn’t work, only to suggest the exact same fragment in its correction…
It also often produces sub-optimal code, it’ll get the job done but might not be the most readable or efficient. This is fine for a starting point and it’s often easy enough to just tweak it yourself but it’s something to be aware of (and a reason I think it’s less helpful for junior devs). It’s the kind of code you end up pushing on the day of a deadline when *anything goes* and you just gotta get it working. Sometimes it tries to make up for this with line-by-line comments explaining the code (maybe in the hopes that you’ll understand and clean it up) but it’s not ideal.
While it can create tests, it only handles the real boilerplate-y part of testing. It’s still important that testing logic is actually led by the developer as testing requires understanding the intent and purpose of the code (which Copilot does not). It can infer some simple unit tests just by looking at the logic inside the production code. There’s a common theme here, but it’s great to use as a starting point. It also doesn’t seem to understand smarter framework features like ParameterizedTests in java or using `it.each` in JS.
At Version 1 we host semi-regular team coding competitions (The Great Java Code Off). As an experiment, I was allowed to be my own team along with Copilot. I wanted to challenge it to take the wheel and be responsible for the majority of the solution (I would only fix compile errors and re-prompt it when its solution didn’t work). While it did give a correct solution (complete with tests), I struggled to understand exactly how its logic worked and it took longer than if I used it in the ways described at the start. In short, it cannot be the brains in your team.
As alluded to at the start, copilot hasn’t done well with our large bespoke repos using in-house frameworks. Its context is very limited and won’t consider other files in the project and while it will learn your coding style for a bit, it’ll lose this knowledge quite quickly. I expect this will change in the future and we’re already seeing other tools like DevGPT claim to solve this problem.
TLDR:
+ It’s often confidently wrong
+ Often does not produce optimised code
+ Makes up for lack of readability for adding comments
+ Lacks wider project context (I expect this to be overcome soon)
Conclusion / Will it replace developers?
No.
…at least not in its current state. It’s a tool to be utilised by developers to increase their productivity. It may eventually replace and reduce the need for as many developers in a single team but we will always need architects and people to steer to the tool. It is not good at big-picture thinking and struggles with multi-step problems. It needs guidance to break the problem down into smaller steps (something that dyslexic thinking is great at).
I wholeheartedly believe that this technology is beneficial and can increase productivity/developer satisfaction if used responsibly. I urge every developer to give it a go, but caution not to blindly trust the code it spits out. Prompting AI is a skill in itself and is important for getting good responses/suggestions. The tech is only going to get better and I’m super excited to see where it goes (even if it does trigger that existential dread in the pit of my stomach).
About the Author:
Jack Reeve is a Full Stack Software Developer here T Version 1.
