Unpacking the promise of developer productivity with Generative AI
Contributors and collaborators: Sumit Awkash, Lakhan Prajapati, Kapil Pant, Jessica Jarvis
Enterprise teams engaged in building applications, AI, and data platforms have undergone a sea change over the past few years with the adoption of agile methodologies, cloud infrastructure, and modern technology stacks. However, there continues to be a persistent quest for speed, efficiency, and quality enhancements to create more value. Generative AI offers a promising solution to enhance developer productivity for such teams by automating tasks, stimulating creativity, and minimizing errors. A recent analysis by OpenAI indicates that after incorporating software and tooling built on top of Large Language Models (LLMs), roughly half of all worker tasks in the US could be completed significantly faster at the same level of quality. Our goal at ZS is to have 100% of our developer community supported by Generative AI toolkits by the end of this year. In this article, we aim to share our learning and some actionable recommendations for improving development team productivity in enterprises.
The low-hanging fruit
AI pair programmers assisting in writing code have become an essential part of conversations around Generative AI for developer productivity. Several solutions are available in the market, ranging from offerings by cloud giants such as GitHub Copilot, AWS Codewhisperer and Google Duet AI, to innovative startups. These tools have shown significant improvements in task completion times for software developers in controlled experiments reported by the creators.
In our own experiments, we have observed efficiency gains of up to 40% for specific coding tasks such as creating DevOps pipelines, building data pipelines for web scraping, and transforming data from PowerBI using Databricks pipelines. However, we did encounter challenges, such as low-quality suggestions, buggy code snippets, outdated and potentially vulnerable library versions, and hallucinations that called on non-existent functions. Nevertheless, these issues were resolved by consolidating best practices and providing adequate training to developers.
It is important to note, however that these gains are often limited to specific, simple tasks that lack generalizability, especially for non-programming activities and personas on teams.
In comes ChatGPT
Tools like ChatGPT cater to a much broader set of activities, going beyond what the current AI pair programmers can do. Several teasers for co-pilots are already out there that promise even greater automation and intelligence for specific tasks. Apart from speeding up tasks, such toolkits also hold the promise for generalists to broaden their range and complete tasks that they needed specialists in the past. A data engineer on our team used a combination of GitHub Copilot and ChatGPT to develop a simple React application 40% faster than an experienced React engineer. This is significant as it demonstrates how engineers can now begin to function as full stack engineers, a role that is hard to fulfil in enterprises.
The implications of such toolkits extend beyond the software development life cycle. Non-technical business users can potentially create the first cut of applications independently, report consumers can perform complex data analyses and predictive modeling using natural language, and embedded testing strategies can strengthen quality of outputs at all stages, leading to safer user acceptance testing (UAT).
Sam Altman, CEO of OpenAI, aptly stated, AI excels at tasks but cannot replace jobs. Automation of various tasks will necessitate job and role reconfigurations within enterprises, for realizing the efficiencies.
What can enterprises do today
It is increasingly clear that beyond the excitement Generative AI has generated, there is potential for it to drive a big change in enterprises. As this field continues to evolve at a blistering pace, several parallel efforts are likely to emerge in all organizations. This presents CIOs and IT leaders with an opportunity to take the reins for ensuring business value is realized with Generative AI in a manner that is scalable, compliant, risk mitigated, and ethical.
Establishing and communicating a strategy to guide the organization’s approach to adoption, whether it is choosing to be early adopters, fast followers, or other segments of the adoption curve, can play a big role in aligning efforts throughout the organization. Tackling data privacy, security and IP protection early is a must and needs cross-functional collaboration. Fostering effective partnerships can help keep pace with the advancements and make informed choices for buying vs. building capabilities.
The change management implications of this technology are tremendous and organized efforts are bound to be needed to shift the mindset. For training the internal muscle, we have seen that choosing established, high functioning teams with sufficiently skilled resources are great. While teams strapped for resources and specialist talent serve as a excellent testing ground to pilot established practices and measure impact before scaling.
Conclusion
Generative AI holds tremendous potential to enhance developer productivity in enterprises. We are starting to see early signs of how this change may come in effect. While AI pair programming tools have demonstrated efficiency gains in specific programming tasks, the combination of specialized software and tooling can significantly accelerate non-programming activities, enabling generalists to take on specialized tasks.
