Copilot and Developer Productivity
Come along as we explain why we chose GitHub Copilot, what our experience has been and where we are headed in light of the recent AI craze
Two weeks ago, on Microsoft’s earnings call, CEO Satya Nadella announced that GitHub Copilot has 1.8 million paying subscribers. But it doesn’t stop there — Copilot’s growth is accelerating. A year ago, 45% of Fortune 100 companies were using GitHub Copilot. Now, it’s 90%.
Since the release of GPT-3, generative AI technologies have shaken up the IT industry. 92 percent of U.S. based developers surveyed at large enterprises report that they are using AI coding tools both in and outside of work. Tools like Copilot promise to turbocharge developer productivity, while recent tech layoffs require companies to do the same amount of work with smaller headcounts.
At Adevinta, we chose Copilot to support our developers in building our marketplaces. In this article, we show you what our experience with Copilot has been and what else we’re doing to leverage the current trends around generative AI.
A Brief History of Copilot
In June 2020, OpenAI released GPT-3, a Large Language Model (LLM) that sparked intrigue in developer communities and beyond. Over at GitHub, this got the wheels turning for a project their engineers had only talked about before: code generation.
The problem with previous attempts at code generation had always been that it was too difficult for the existing models. GPT-3 changed that — suddenly there was a model good enough to ask coding questions and receive sensible answers. GitHub Next, the research and development team at GitHub, evaluated the model with crowdsourced self-contained problems. Soon, the model was solving upwards of 90 percent of the problems.
The first concept was a chatbot that the team would ask questions and receive runnable code snippets. They thought of a better way to integrate this process into a development environment and came up with the plugins known as Github Copilot and GitHub Copilot Chat.
We opted for Copilot due to our previously close relationship with GitHub and the ease with which we could get started from a product and billing perspective. The idea was to slowly add Copilot licences to our workforce and see how our developers progressed with it. Later in the article, we will explain our launch process in more detail.
Additionally, Gergely Orosz from the Pragmatic Engineer newsletter has provided a great overview of Github Copilot and ChatGPT alternatives which was tremendously helpful in understanding our options in this space.
About Us
In the Cloud & Infrastructure organisation, we offer a suite of platform and data products to developers across all of Adevinta’s marketplaces. With a workforce of around 3000 employees working in Product & Tech, spread across multiple brands, countries and languages, our goal is to meet all their needs and preferences.
The launch of GitHub Copilot at Adevinta has been overseen by a small interdisciplinary team. Together we evaluated Copilot from multiple angles such as security and legal, far beyond our standard purchasing process. Generative AI is a special case in these aspects and we wanted to get it right.
Following our evaluation, we managed the launch, adoption and usage of Copilot at Adevinta. Our goal is to empower developers with this novel technology and enable them to build better software, including better products for the buyers and sellers on Adevinta’s marketplaces.
Taking Off
In late 2023, we launched Copilot to a small group of beta users. We recruited approximately 100 developers who were keen on being among the first to use this technology at Adevinta.
At this time, GitHub did not offer usage statistics for Copilot. We had to find another way to monitor how our developers were doing. Our plan was to have a diverse group of engineers (nationality, discipline, demographics etc.) that we would survey to understand the impact Copilot had on their daily work. After a couple of weeks, we asked them how things were going with their new coding buddy.
The survey results gave us confidence that our developers appreciated this new technology in their daily work. In the following weeks, we designed a self-service process for Adevintans to request their own Copilot licence. Our internal “Copilot Community” grew quickly to hundreds of users and to accommodate everyone we opted into all beta programs offered by GitHub. (Copilot Chat was not generally available for JetBrains IDEs until March 2024)
Over the next few months, we saw tremendous growth in the number of licences being assigned. Most of the time we allowed assigned licences to grow organically. We didn’t make a company-wide announcement about the service until mid-February when we hosted an internal event in collaboration with GitHub at our Barcelona office. The event was also live streamed and we had Adevintans join from all over the globe.
During this event we had a hackathon that required attendees, among other things, to solve problems with the help of Copilot. The goal of the event was to increase the internal exposure of Copilot to our developers. We were happy to see a significant increase in issued licences during the event.
At the time of writing this article, GitHub made their “GitHub Copilot Usage REST API [alpha]” available. This API gives us important information about the actual activity of our users, as well as breakdowns of suggested lines of code and their development environment.
(Edit April 2024: GitHub has officially launched the public beta for Copilot metrics)
The biggest takeaway for us was the amount of dormant licences. When we looked into the activity logs in detail, we noticed that most activity comes from developers. Engineering Managers, Product Owners and non-technical roles were the most inactive groups. Using this observation, we’re looking into deactivating licences automatically when they haven’t been used for 30 days.
The other interesting detail we gleaned from the data is that the average acceptance rate for Copilot suggestions sits around 20%. If you have ever used Copilot you will know that it can suggest code quite often. The fact that every fifth suggestion is accepted by our developers is not insignificant. However, it’s far away from the idea that AIs can or will replace developers entirely.
Keeping Aloft
A couple of months have passed since making GitHub Copilot available to all developers at Adevinta. To keep a pulse on what our developers think of generative AI and how they use it in their everyday work we use our recently developed Developer Experience survey.
The productivity impact of Copilot should not be measured on accepted lines of code or active users alone. We believe it’s important to check in with our developers regularly in order to get a sense of the metrics that are not reported by GitHub. With the Developer Experience survey, we try to understand multiple aspects of productivity, including perceived productivity boosts from the usage of Copilot and quality of code suggestions.
“What generative AI tools have you used most for work?”
“How would you rate the quality of code suggestions?” (1 = poorly; 5 = very good)
We learned a couple of things from this survey. The first is that GitHub Copilot is not the only generative AI solution our developers use. Copilot’s biggest competitor however seems to be “not using generative AI for work”. As previously mentioned, we see a steady climb in active users already, so we’re positive that this will change for the better.
The other news is that developers seem to be indifferent to the quality of suggested code by Copilot. From internal conversations, we know that suggestions are mostly a hit or miss. They tend to be most effective for tedious work, but fall short on contextual, business-logic related code. All things considered, we can still see developers favouring Copilot, with almost half of them reporting that Copilot accelerates their workflow.
Cruising Altitude
We believe that Copilot is a tool that is as smart as the engineer using it. It seems to be most beneficial to those that need it the least and least beneficial to those that need it the most. We’ve seen (junior) developers put too much trust in Copilot and then having to ask for help from senior colleagues. This isn’t an issue unique to Copilot, because previously nothing stopped them from copy-pasting solutions from other sources like Stack Overflow (SO) either. The only benefit SO has over Copilot in this sense, is the extra friction of copy-pasting code. Perhaps it just raises the bar for code quality and companies need to ensure they’re not falling behind technically.
For us, Copilot looks like it’s going to stay. Similar to a developer’s favourite IDE, it’s just a tool to accomplish the task at hand. Taking it away from our developers would result in a small uprising, because most report a net-positive impact on their daily work in our Developer Experience surveys.
Based on our findings, we encourage everyone to not become over-reliant on generative AI models and use their best judgement when working with them.
