Can Github Copilot Change The Programming Landscape?
First Glance at Github Copilot
How it works, its benefits and limitations, and should you be worried as a programmer?
Github turned many heads when it announced Copilot in June 2021.
What is Copilot
Github describes Copilot as a Your AI pair programmer. It is currently in beta and invite-only mode.
It offered a glimpse of what coding may look like in the future and made many folks feel insecure. Will developers become obsolete? Will AI take over our jobs as programmers? These questions naturally came up, but as with most tech stuff, there’s no simple answer.
I was excited to take a crack at it after I recently got my invite. In the following sections, I will test the Copilot with a series of programming challenges, from our hiring screening tests to some specific back-end and front-end problems.
Screening Challenge
I will start the Copilot challenge with screening tests that we typically send to junior programmers. These are general LeetCode style algorithm development problems where the developer can choose any programming language to develop a solution.
First Problem: String Manipulation
Complexity: Simple
Have the function DashInsert(str) insert dashes (‘-’) between each two odd numbers in str. For example: if str is 454793 the output should be 4547–9–3. Don’t count zero as an odd number.
Let’s get started. We have a blank file in Visual Studio Code, and we choose Typescript as our language. As we start typing the problem, we see auto-complete suggestions for the function name, and as we proceed and accept, we see further recommendations for the rest of the function.
Generated code below👇
Time to put the Copilot code to the test 💪
Bingo. All tests passed like a charm, so far, so good!
However, on a review, we see one potential issue. The loop may go out of range if the argument is accurately typed as a String. With its current implementation as any, it may give erroneous results for corner cases.
Second Problem: Array Manipulation
Complexity: Medium
Have the function NearestSmallerValues(arr) take the array of integers stored in arr, and for each element in the list, search all the previous values for the nearest element that is smaller than (or equal to) the current element and create a new list from these numbers. If there is no element before a certain position that is smaller, input a -1. For example: if arr is [5, 2, 8, 3, 9, 12] then the nearest smaller values list is [-1, -1, 2, 2, 3, 9].
Resulting code 👇
Time to test!
FAIL! The Copilot made an assumption and returned an array, but the test case was expecting a string. The question didn’t specify this, so we give it the benefit of the doubt. We should have a quick fix; let us reformat and return the array as a space-delimited string using this change.
return result.join(“ ”);
Almost there, this feels like actual programming😅
One test case still failed. On closer inspection of the code, we find that it failed the check for the nearest element where the question clearly stated smaller than (or equal to).
So, we add a quick fix by changing the condition to include the equality check.
And we’re done, Copilot has passed our screening challenges🎉
Back-end Challenge
First, let’s try a couple of things with writing data-fetching queries.
1. Get all users from an SQL table that have age above 18
2. Get all users from a MongoDB collection that have logged into the system in the last week
The code 👇
Feedback: I ran this past two rockstar Senior Developers at our company, Talha and Danyal. Here’s the feedback:
“The code seems to have no logical issues, but async/await implementation with typings would’ve been better and more upto the latest standards”
Next, let us write a simple function to get the current Bitcoin price. Our language of choice this time is Python:
Write a function to get current Bitcoin price
Generated code 👇
Senior Engineer Feedback:
“Code looks precise and accurate but it’s better to take the import statements out of the function.”
Last challenge on the back-end. Let us test how it works for setting up a subscription with Stripe API, an everyday use case for apps. We will use Java for this one.
Create a Stripe subscription for the given user and plan
Copilot suggestion 👇
Senior Engineer Feedback:
“Seems to have recursive infinite call on createSubscription. Would be good to be able to run and test this code and improve it from there.”
Front-end Challenge
We will test Copilot with DOM manipulation and have it perform some image styling.
Find all images without alternate text and give them a red border
Generated code 👇
Front-end Engineer Feedback:
“Code looks good and to the point! We also ran this in browser console to see it working.”
Summarizing Results
- Accuracy of solution: Most of Copilot’s algorithms were accurate, given that the problem statement had sufficient information. Where not, it took assumptions and did have a hiccup on a corner case.
- Performance of solution: Although I didn’t share the performance results in this article, we generally found the code to be efficient.
- Speed of coding: I think this is where Copilot does add value. Whereas coding is just one of many important jobs that an engineer does, Copilot can help save some time by spending lesser time. Engineers can save time on building an algorithm and reading documentation. Mind you, it should also add some extra time to understand and test the code.
- Risks: Bugs can creep into the code if not reviewed and adequately tested.
- Code quality: Engineering teams are picky about their code and need consistency across the codebase. So, keep some cushion to refactor the codebase after testing the solution.
This space is getting exciting, and in my view, many engineers can benefit by adding Copilot to their repertoire of skills.
My Key takeaway? Coding faster alone can not make you a better software engineer. Understanding the end-users, helping define the problems, designing a good solution, working on an efficient architecture, and abstraction differentiate you from the rest!
