Louder than a Whisper: Praise for AI Coding Assistants That Doesn’t Overpromise
In pursuit of developer happiness and disruptive speed, companies are considering code generators like Amazon CodeWhisperer (now Amazon Q Developer). What results are realistic?
Software developers are in particularly high demand today. The unemployment rate for tech jobs is well below the national unemployment figure and 78% of employers in IT report difficulty finding talent with the right skills.
For developers, we have more good news on job security: AI-powered code generators will not replace you.
For employers, we have a word of caution: AI-powered code generators will not replace your developers.
Most, if not all, of Slalom’s data and AI teams are spending many of their working hours in conversations and engagements focused on generative AI. We would call this a hype cycle if not for our firsthand experience observing how generative AI is going to change the lives of people everywhere, including the ways we work.
The change we see isn’t layoffs — far from it. Take the word of Joshua Gruber, a senior technology leader at Slalom Build who has delivered multiple workshops exploring the potential of AI coding assistants including Amazon CodeWhisperer (now Amazon Q Developer):
“Most development teams have a ‘two pairs of eyes’ rule for human-generated code, whether that comes from pair programming or code reviews. For a long time, we will need to take at least this same level of care with code produced by AI.”
Gruber’s article about AI coding assistants is a great resource for developers and technology leaders who are considering implementing these tools but aren’t sure which tool to use. If you’re curious about Amazon CodeWhisperer, keep reading for more about what to expect.
What’s different about Amazon CodeWhisperer?
On the day Amazon Web Services (AWS) announced the general availability of CodeWhisperer, AWS published a blog post written by Swami Sivasubramanian, who oversees database, analytics, and AI and machine learning services at AWS.
Sivasubramanian astutely described the challenge(s) that tools like CodeWhisperer address:
“Software developers today spend a significant amount of their time writing code that is pretty straightforward and undifferentiated. They also spend a lot of time trying to keep up with a complex and ever-changing tool and technology landscape. All of this leaves developers less time to develop new, innovative capabilities and services. Developers try to overcome this by copying and modifying code snippets from the web, which can result in inadvertently copying code that doesn’t work, contains security vulnerabilities, or doesn’t track usage of open source software. And, ultimately, searching and copying still takes time away from the good stuff.”
What Sivasubramanian describes are challenges that every AI coding assistant on the market today — as well as more general-purpose tools like ChatGPT — aims to eliminate. But coding assistants tend to be different from general-purpose AI in a couple key ways:
- Coding assistants are, by definition, designed specifically to help write code
- Coding assistants integrate directly with IDEs (integrated development environments), which allows them to make continuous code suggestions while developers are coding, not just in response to natural language prompts
- Most coding assistants can track their suggestions in developers’ code so that developers can always identify what came from the tool
- Most coding assistants can identify code in their suggestions that might be referencing open source training data (e.g., publicly available code repositories) and cite the applicable license
- Coding assistants adapt to individual coding styles in structure and syntax
In addition to these more standard capabilities of AI coding assistants, CodeWhisperer has a few features that are more unique:
- CodeWhisperer’s suggestions are based on the developer’s immediate comments and code as well as contents of other open files in their IDE
- CodeWhisperer delivers “optimized” suggestions when using AWS APIs such as Amazon Elastic Compute Cloud (Amazon EC2), AWS Lambda, and Amazon Simple Storage Service (S3)
- CodeWhisperer has a setting for filtering out what might be open source code from its suggestions, if desired
- CodeWhisperer offers on-demand security scans that identify security vulnerabilities in your code (50 scans per user, per month)
How fast is Amazon CodeWhisperer?
If you’re asking yourself this question, let us remind you: AI-powered code generators will not replace developers! So, the better question is: How fast does Amazon CodeWhisperer help developers code? The answer is 57% faster, according to AWS research conducted during CodeWhisperer’s preview period. Ever the realistic optimists, we at Slalom usually set expectations of between 30% and 50% faster development across AI coding assistants, but we expect that figure to get better the more the tools learn and improve, and the more that organizations understand how to implement them (more on that in a minute).
Interestingly, we are starting to observe in our work helping customers implement and experiment with AI coding assistants that different levels of engineers tend to have different experiences using the tools, with less experienced developers moving much faster with the help of coding assistants. While we can’t say for sure why this is, it resembles generative AI’s impact in other disciplines, such as writing.
It brings to mind another question: How fast is too fast? James Grunewald, a software engineer at Slalom Build, has heard many versions of that question in his work with customers. Grunewald says:
“There was a lot of concern in some of those rooms about how less experienced devs could use [AI for coding] to be sort of uncritical. It’s another tool that allows them to go faster, but perhaps too fast, or without enough diligence.”
That’s why it’s so important when implementing AI coding assistants to factor in both speed-related metrics like speed to market and sprint velocity, as well as quality-related metrics like the number of bugs in AI-supported code. If you’re getting more bugs, that’s obviously a problem! It doesn’t mean you should stop using these tools altogether, but it might indicate that your organization needs to pair your use of coding assistants more closely with development best practices. Gruber (Slalom Build technology leader) emphasizes Agile methodology and CI/CD pipelines as two such best practices.
How do I get the most out of Amazon CodeWhisperer and other AI coding assistants?
We’ll give it to you straight: Getting to 30%, 50%, or 57% faster development — while maintaining code quality — takes finesse, especially if your goal is to do so at enterprise scale. Our top advice for organizations echoes a lot of what we’ve already discussed:
- Get clear on your metrics. Look at speed-related metrics while keeping a watchful eye on your bug count and the complexity of your code. Make sure everyone involved in the work understands those metrics and your plan for tracking them.
- Incorporate best practices. Protect against code quality deterioration by incorporating best practices like Agile methodology and CI/CD (continuous integration, continuous delivery) pipelines. For the record, our proprietary engineering approach at Slalom and Slalom Build has best practices like these built in — learn more about our product engineering methodology.
- Be deliberate with your prompts. To quote Gruber again, “Thinking through what you were trying to code, writing that down, and feeding that into CodeWhisperer as a comment is much more powerful than giving it a little bit of an idea and then editing it or iterating on it currently.”
We’ll end our advice with what is probably both the hardest to measure and most important part of AI coding assistants: developer happiness. A “fast” developer is far less valuable in an organization — and less likely to stay with an organization when their skills are in such high demand — if they aren’t also satisfied and engaged. It’s easy to assume that using AI to leave behind more of the mundane parts of their jobs than ever before imaginable will make developers happier. But never assume someone’s happy. Instead, ask them. When you implement AI coding assistants like CodeWhisperer, don’t skimp on retrospectives and team surveys that give developers an opportunity to share their experiences using these tools.
What happens now?
Picture a world where tools like Amazon CodeWhisperer have sped up the “busy work” of development enough to allow engineers to spend most of their time doing differentiated, creative work. They are reaching and maintaining longer “flow states” working on things that are more directly related to the fundamental problems companies are trying to solve: customers’ problems.
When you can spend more of your time solving customers’ problems instead of writing boilerplate sections of code, sifting through different software libraries, and customizing code snippets found online, your sense for strategy gets more acute. You realize what you need to be doing to help your customers in a unique, exceptional way that other companies in your industry haven’t considered or deemed possible. And, you have an easier time pursuing that.
So, we’ll leave you with one last question: With the time your organization could save from using AI coding assistants, what will you do not more of, but differently? What will you build that you’ve never thought of building before because you and your team didn’t have the bandwidth or weren’t in the right headspace? That’s the question we think everyone should be asking. Yes, let’s talk about developer productivity. And then, let’s talk about the “impossible” things we can achieve next.
PS: AI-enhanced engineering with Amazon CodeWhisperer is the focus of one in a collection of generative AI workshops we’ve designed with AWS users in mind. Explore all our generative AI workshops for AWS.
PPS: Want to take a closer look under the hood of CodeWhisperer? We recommend starting with the 4-minute overview from AWS, then checking out a demo of CodeWhisperer’s recently announced integration with AWS Glue Studio.
Slalom is a global consulting firm that helps people and organizations dream bigger, move faster, and build better tomorrows for all. Learn more and reach out today.
