Has GitHub Copilot genuinely managed to be a companion?
It was the month of June’21, we heard the news of launching a new AI-powered pair programmer by GitHub, which would essentially collaborate with our projects and write the code for us, feels astonishing right! It generates quite a stir. I was fortunate enough to try out the early access of GitHub Copilot. It’s been one year since then. In June’22, GitHub announced to make Copilot available publicly with a charge of $10/month but free for students & open-source projects.
This blog features a detailed review of my experience with Copilot. Many times it made my jaw drop; it made me feel ‘whoa’ like I’ve harnessed the power of something mystical which isn’t normal from a developing tool. As well as, it made me worried about the blind approach and mechanism Copilot works on.
What is it Actually
Copilot is an AI-powered tool developed by GitHub in collaboration with OpenAI, a San Francisco-based AI firm heavily backed by Microsoft. It provides code suggestions based on the comment and context of code blocks. It can suggest an entire function just by reading your function name or the comments you add. It can automatically detect the relations within your defined variable, automate repetitive code, or create unit tests that too, adopt your own coding style. In simple terms, it is relatively the same as Gmail suggests next texts as you continue typing an email, Copilot suggests code in quite the same manner.
Copilot mostly supports most active current programming languages, but it works best on Python, JavaScript, TypeScript, Ruby & Go. And, the language is detected automatically. It can also determine if a file is using multiple languages combined. For example, in an html file it suggests Html template but as soon as you’re within those script tags, it effectively switches to JavaScript.
So, What works under the hood?
Copilot is powered by Codex, a modified and production version of GPT3 or Generative Pre-trained Transformer, a third-generation very large language model. GPT3 is built on a deep neural network trained with 45TB of text corpus. It has 175B parameters, which is the largest till date. The model is mostly good in English language and smart in predicting the next word sequence with a fluent human-like text.
It creates max 10 suggestions at a time. The precision of suggestions improves as you type more. The Copilot editor extension sends your code to the GitHub Copilot server as you type, then it uses OpenAI codex to synthesize and recommend suggestions to you. Copilot is trained on open-sourced, publicly available repo and is getting better every day by analyzing the suggestions that are approved.
Copilot in Action
Before we dive into the significance of the title of the blog, let me show you some of the experiments to better understand the practical aspects of Copilot.
It has tried to guess the most possible word as I type ‘password’, first it creates the suggestion for password_hash but as soon as I continue typing by ‘_ge’ it changes the suggestion to password_generator. It also finds out that we don’t have the required packages imported, so locally import random & string too. But does it behaves the same always? Let me do the same action once again…
It is so random in nature, that repeating the same task in the same file generates random results. More than that, this time Copilot fails to detect and add the modules like string & random without even importing. More like this can be seen below,
Again, it pushed a module without import. Now, let me instruct my companion to write some code, as I said.
Well, my companion has definitely attended his high school classes. Let’s test him if he remembers anything on currency conversion…
Hmm, it automatically used a wide exchange rate of 65 which was from 2018. That pretty much explains his training periods. Now, I’ve tried to run Copilot inside a Flask project.
It efficiently understands that I want to import blueprints, and already have explored my projects and imported the only blueprints that I have. It is also effective at reading the context of class and the methods defined.
I’ve also asked it to scrap the amazon website and get the best price for products…
Though it was confused on the first try, after changing the function name it writes some code but that won’t do the job. Furthermore, I’ve learned that it can effectively do a binary search, bubble sort, perform matrix multiplication, have an idea of complex derivative formulas, summarize Wikipedia pages, write a tic-tac-toe game, etc. Using Copilot was nonetheless fun, at least when Copilot is demanding of being a Human, LOL!
Can it become Sentient and took over dev jobs?
A big Nooo…
I’m not sold on the idea that developer jobs could become extinct. Still understanding the risk associated or the impact of events, to know the engineering constraint or architecting solutions is something that belongs to humans. There are always some intuitions which impact more than just metrics. But, it is just the beginning and Copilot has set a benchmark in its field, there’s more coming to level up the benchmark metrics. But it is for sure that in the future automation is going to set an impact in every industry and a category of people will be there for suffering. Our IT industry isn’t out of the radar. It may happen that in future, data entry or similar jobs which can be simulated with a rule-based approach, can be on the verge of extinction. Engineers may spend most of their time just reviewing, checking the integrity and writing test cases rather than just developing, who knows!
So, can we call it a companion?
Copilot was built with an aim of helping programmers write code less and fast while maintaining the standard of coding. But it is not the first of its kind, TabNine & Kite is already providing AI-powered code completion for a few years from now. These are also using a model trained on the deep neural network but are only able to suggest the next one or two lines. The difference is Copilot’s ability to generate multi-line code or even documentation, just by only reading the context of code.
It always can be argued about the point of using programming to automate the programming itself, but what truly amazes me is the promising technology, it’s of immense interest to observe what others revolutionary events are awaiting us in future. GPT3 model holds remarkable revolutionary power. Copilot is an extremely fun and easy-to-use tool. Yes, I’d prefer to call it an incredibly smart tool waiting for following orders rather than a ‘companion’. To me, a companion is someone (or something at least in our case) who truly guides us while writing code or building the logic along with us. But instead of doing this, Copilot blindly assume what we were doing is absolutely correct, and started following us. For a complex application, if we develop a fallacious belief Copilot still encourages us and provide suggestion to write less and fast code!
What more could possibly go wrong?
According to GitHub, the suggestions provided by Copilot are not guaranteed to provide ‘the most optimal’ solution. One can even encounter security vulnerabilities, deprecated code or code blocks with already bugs. According to a study in 2021, Copilot generated 1692 programs of which nearly 40% has security threats. There’s a prominent reason behind this. As we know Copilot has been trained publicly available open-sourced projects, which includes projects written by the world's best programmers and maintained by tech giants like Google, Microsoft, Amazon and more, this list not going to stop anytime soon. But we should keep in mind how a language model works, it tends to follow what most people write, on average. Most of these projects are itself neither bug-free nor actively maintained for a couple of years. In fact we, the human programmers, do know the fact that bugs and issues are inevitable in any system. Copilot put its best effort to suggest what those ‘average’ quality codes could’ve been if you were given to write it out. What matters more, it doesn’t even try to compile it or at least checks for any deprecated or missing packages used in its suggestion.
One of the common tasks for us is to refactor or maintain the existing project, avoid depreciation of existing used packages, and keep the codes up-to-date in order to have it secured for time being. This is where Copilot lacks, it knows how to get the things done but doesn’t have an idea of the impact or updated ways of doing things, well at least for now.
Impacts of Automation on the domain of Automation itself
If I were to concentrate on the impact of Copilot in my domain of Data Science, we’ve no particular way of doing things. We need to try out and sort things based on the current performance we’re achieving.
Essentially, having the knowledge of repo based on machine learning and deep learning too Copilot is able to perform almost all known & common tasks like normalization, model training or even cross-validate and plot metrics. But it throws me into problems like, it generates a function to perform cross-validation using halving grid search but the params it chooses automatically result in poor performance.
One other backlash that I’ve faced is in my early days of learning ml where I’ve found my baseline regression model score is like below,
After being exhausted for about 3 hours, I discovered the below function,
Copilot is giving me immense fun, I was plotting the distributions with help of auto-generated suggestions. For the above function, in line no 3 I’ve created a new variable but in line no 5 it surprisingly log-transformed the original filtered_df dataset and I missed that causing me a lesson in my career.
Most of the time it surprises me but in a sensitive context or a project that is not so well organised I choose to turn Copilot off. Another reason behind this is I’m writing code and it just popped up with its suggestion interrupting my flows. It’s seriously perfect for generating small pieces of code or simple logical helper functions but we should never blindly trust the code it spits even if it is as easy as adding up two variables (especially if you’re on JS, 2+2 may give you 22).
There’s more to Criticize?
Yes Indeed, GitHub claimed to not share your private codebase. But it faces open-source code laundering and copyright infringement.
The open-sourced code hosted on GitHub with an MIT License allows redistribution but requires giving credit to the owners. Although one debate can be worth starting here, when you host your code on GitHub, you’re giving the right to GitHub authority to use your code for the improvement of their products and feature, according to their Terms of Service. But the problem that arrives when you accept its solution, results in your unintentional copyright infringement as GitHub has the right to reuse the code but you don’t.
Copilot is a state-of-the-art, fine-edged nifty product that sparked a lot of debate and holds the potential for the long run. It may provide you with results from tons of data, but it fails to explain why he suggests so. I believe the explanations I hold on the projects I’ve done, are the key to propelling my career. One should definitely try it out and for the sake of innovation, it is appreciated.
To the students, Copilot can easily complete all of your college assignments or even interviews. But rather than cheating, you should use it to learn how a single problem can be solved with 10 different approaches, try to gather knowledge using the technology and don’t let technology overwhelms you.
For the developers, by no means Copilot is perfect but it can definitely boost your production by taking over repetitive tasks. But it’s worth noting that Copilot should be used with caution. You should only use it when you know what you’re doing and definitely review the suggested code before accepting. It is not recommended to use it to train a model, generate passwords or create database dependency or in a sensitive, abstract or not-so-well-organized project, or any concept that you’ll not able to decipher.
Copilot is not a replacement in our helpful community; I’d still prefer to search in StackOverflow and read through all the answers and comments but it may guide me to my solution with a ‘teaching-with-references’ approach. But it is for sure the sharp-edged automation that we’re about to see in the coming years, is going to shape how industry replaces us.
So, here I’m wrapping it up. If you’ve come so far, I’m glad to let you know it’s my first tech blog actually. Please do share your valuable suggestions, I’d be much appreciated to work on those. Also, let me know if I’ve missed anything.
I’d love to connect with you via LinkedIn
