Quantifying the Productivity Gains of Generative AI for Developers
by Jerry Cuomo with support from @Haechul Shin, @Blaine Dolph and @Chris Hogen | IBM 2023
Generative AI models like GPT and LLaMA are transforming the software development process and providing new ways to enhance developer productivity and deliver value to customers. In this paper, we examine the role of generative AI models in enhancing productivity across a range of developer experience levels, from novice to expert.
The productivity model presented in this paper examines the potential gains for developers of varying experience levels through the automation of repetitive tasks, optimization suggestions, and decision-making assistance provided by AI. Our analysis estimates potential gains approaching 70% over a five-year period. The paper also outlines a phased approach for measuring these productivity gains.
The productivity gains enabled by the introduction of generative AI allow for a shift to value and create opportunities for increased innovation, improved quality, enhanced customer experience, and faster time-to-market. Specifically, we estimate that generative AI can lead to a 15–20% increase in the number of new products or features developed, a 10–15% reduction in the number of bugs found in production, a 5–10% increase in customer retention and loyalty, and a 10–15% reduction in time-to-market for new products or features.
Finally, we argue that pricing models must be reevaluated to reflect the actual value delivered to the client, rather than being based on billable hours. We recommend value-based pricing or performance-based pricing models that align incentives for both the client and the provider, which can incentivize the provider to deliver greater productivity gains and create better profits while improving productivity for developers.
Overall, the introduction of generative AI represents a significant opportunity for businesses to improve developer productivity, drive growth, and deliver greater value to customers. By adopting the right pricing models and leveraging AI assistance effectively, businesses can unlock even greater productivity gains and position themselves for success in a rapidly evolving digital landscape.
Developer Groups and AI Assistance
Developing software is a complex task that requires a diverse team of engineers with specialized skills and experience. For our productivity model, we have chosen to focus on four skill levels in development: junior, mid-level, senior, and technical leads/architects. For each skill level, we will include the following information:
- Typical years of experience and estimated salaries as of 2021
- Description of the role and typical tasks performed
- Proportion of time that may be required for a digital transformation project
- How AI can provide assistance to developers in this role
Junior developers:
- Have 0–2 years of experience
- Estimated salary: $55,000 — $80,000 per year
Junior developers may make up a small percentage of the team, typically around 10–20%. They may be responsible for developing basic features, fixing bugs, and assisting mid-level and senior developers with more complex tasks.
AI can provide guidance and suggestions, allowing junior developers to learn and grow in their roles. For example, AI can help junior developers identify syntax errors, suggest code snippets, and automate repetitive testing tasks.
Mid-level developers:
- Have 2–5 years of experience
- Estimated salary: $80,000 — $120,000 per year
Mid-level developers may make up the largest percentage of the team, typically around 50–60%. They may be responsible for developing core functionality, integrating different systems and components, and working with senior developers and technical leads to ensure the architecture and design meet project requirements.
AI can provide optimization suggestions and help mid-level developers save time on routine tasks. For example, AI can help mid-level developers optimize code for performance and scalability, automate build and deployment tasks, and identify potential bugs and errors before they become problems.
Senior developers:
- Have 5–10+ years of experience
- Estimated salary: $120,000 — $175,000 per year
Senior developers may make up a smaller percentage of the team, typically around 15–25%. They may be responsible for designing and developing critical features, optimizing the system for performance and scalability, and mentoring mid-level and junior developers.
AI can help senior developers identify potential issues and suggest improvements to the code-base. For example, AI can help senior developers identify security vulnerabilities, suggest refactoring or optimization opportunities, and automate routine testing and deployment tasks.
Technical leads/architects:
- Have 10+ years of experience
- Estimated salary: $175,000 — $250,000 per year
Technical leads and architects may make up a small percentage of the team, typically around 5–10%. They may be responsible for defining the system architecture, guiding the development strategy, and collaborating with stakeholders to ensure the system meets business requirements.
AI can help technical leads/architects identify potential issues and suggest improvements to the software architecture. For example, AI can help technical leads/architects identify potential scalability or performance issues, suggest design patterns or best practices, and automate code review and analysis tasks.
By using AI to assist developers in each of these groups, we can optimize their productivity and help them learn and grow in their roles. AI can help junior developers build their foundational skills, assist mid-level developers in becoming more effective team leaders, and help senior developers stay up-to-date with the latest trends and techniques. Additionally, AI can assist technical leads and architects in making better decisions for their projects and ensuring that they align with business goals and objectives.
Productivity Model
To quantify the potential productivity gains of generative AI in assisting developers, we have developed a model that builds on the thesis that productivity gains are best realized across a set of skill groups over a period of time, in this case, five years.
It is important to note that the generative AI needs to be trained and fine-tuned against specific practices that factor a company’s IT environment. Higher skilled developers are assumed to have the best experience with the existing environment and are, therefore, best suited to train the AI models. The effectiveness of training diminishes as we approach the junior developer group, which is assumed not to participate in training. To demonstrate the model’s effectiveness, we utilized a standard five-year plan for the transformation of an insurance company’s claims processing system.
Introducing generative AI can lead to productivity improvements across different developer groups. Estimating the exact productivity improvement can be challenging, therefore, requires careful consideration of several factors. The author provide a calculation spreadsheet for the productivity gains upon request.
The following table illustrates the anticipated productivity gains for each developer role during the five-year time-frame. It is important to note that the total productivity gain across all developers is weighted based on the role distribution within a project. For instance, considering that mid-level developers typically constitute 50–60% of the team, their productivity gains have a greater impact on the overall productivity gain calculation.
All roles make steady progress over the 5-year period, with junior developers experiencing significant growth in first year. By the fifth year, all roles are steadily approaching or surpassing the 60% threshold in terms of productivity gains. Now let’s break down the changes happening in year one, three, and five that are driving the productivity gains.
Year 1: In the initial year, all roles experience productivity gains. Junior developers stand out with an impressive 14% increase, almost twice the gains of other roles. This highlights the effectiveness of generative AI in supporting junior developers, enabling them to approach their tasks more efficiently and further develop their skills.
Year 3: All roles demonstrate significant productivity gains, with junior developers showing remarkable improvement in tasks such as creating code unit tests. Mid-level developers also experience substantial progress, benefiting from increased efficiency and expanding expertise. The contributions of AI are particularly evident for senior developers and architects, enabling them to excel in code and security reviews, evaluating new technology, guiding technical strategies, and ensuring high code quality.
Year 5: The integration of AI has become well-established across all roles by Year 5, with varying growth rates. While junior developers maintain strong progress, mid-level developers face the challenge of balancing their skill level with the complexity of their work. Nonetheless, AI assistance continues to be instrumental in enhancing productivity and supporting developers in their tasks, enabling sustained gains throughout the 5-year period.
It’s worth noting that the actual productivity gains will almost certainly vary based on several factors such as the specific use cases, the capabilities of the generative AI, and the level of expertise and training of the developers. It’s also important to consider the potential ethical, legal, and societal implications of introducing AI, and to ensure that the AI is designed and deployed in a responsible and transparent manner.
Opportunities for Growth and Value Delivery
The introduction of generative AI can create opportunities for growth and value delivery beyond simply increasing productivity. Some potential opportunities include:
Increased Innovation: By automating repetitive tasks and freeing up time for more creative work, developers can focus on developing new and innovative solutions to business problems. This could lead to an estimated 15–20% increase in the number of new products or features developed. See DEEPCODER: LEARNING TO WRITE PROGRAMS
Improved Quality: With the assistance of AI, developers can more easily identify and fix bugs, resulting in higher quality software. This could lead to an estimated 10–15% reduction in the number of bugs found in production. See DeepCode: Feedback generation for automatic bug fixing
Enhanced Customer Experience: AI can help personalize and streamline customer interactions, leading to higher levels of customer satisfaction. This could result in an estimated 5–10% increase in customer retention and loyalty.
Faster Time-to-Market: By automating tasks and improving efficiency, AI can help reduce the time it takes to develop and launch new products or features. This could lead to an estimated 10–15% reduction in time-to-market for new products or features. See Accelerating the Software Development Lifecycle with Artificial Intelligence. See Driving Software Delivery Automation With AI/ML
It’s important to note that these estimates are based on the assumption that the productivity gains outlined in the model are realized. The actual impact on innovation, quality, customer experience, and time-to-market may vary depending on the specific project and business context.
By viewing the introduction of generative AI as an opportunity for growth and value delivery rather than solely as a cost-saving exercise, businesses can take advantage of the full potential of AI to drive innovation, improve quality, enhance customer experience, and accelerate time-to-market.
Measuring Productivity:
While measuring productivity gains in software development can be challenging, it is essential to evaluate the impact of AI-assisted development tools. It is recommended to take a phased approach to implementing productivity measurement metrics over a three-year period. Here is an example of a three-year plan:
Year 1 — Survey and Instrument: In the first year, start by conducting a survey of developers to gather their opinions on the impact of AI-assisted development tools on their productivity. This survey can ask developers to rate their productivity before and after the implementation of AI tools, as well as their experience with the tools. Additionally, start tracking basic metrics such as bug counts, number of features delivered, and time to market for projects that use AI tools versus those that don’t.
Year 2 — Instrument and Automate: In the second year, continue to survey developers, but also begin to implement more comprehensive productivity measurement tools. For example, start tracking code quality metrics such as code complexity and code coverage, as well as more detailed bug tracking data such as the number of bugs found and fixed, and the time it takes to fix them. Additionally, track more detailed project management metrics such as the time it takes to complete a project, and the percentage of projects completed on time.
Year 3 — Automate and Iterate: In the third year, continue to track all of the metrics from the previous years, but also start to automate the measurement process as much as possible. Implement tools such as automated code review and testing to generate data on code quality and bug counts. Additionally, use project management tools such as agile methodologies to track the progress of projects and generate data on time to market and project completion rates.
By following this phased approach, it is possible to develop a comprehensive productivity measurement model that can accurately evaluate the impact of AI-assisted development tools on productivity gains over time.
Pricing Considerations and Recommendations
When it comes to pricing projects that involve the use of generative AI and other productivity tools, it’s important to consider both the cost savings and the potential value that can be generated. One pricing model that should be avoided is labor rate based pricing, as this can lead to a focus on billable hours rather than the actual value delivered to the client.
Instead, pricing should be based on a value-based approach that takes into account the potential cost savings and value generated for the client. This could involve a fixed-price or outcome-based pricing model, where the price is based on the results achieved rather than the number of hours worked. For example, if generative AI is expected to reduce the time-to-market for a project by 20%, the pricing can be based on the value of this time savings to the client. See Value-Based Pricing For New Software Products: Strategy Insights for Developers
Another consideration is the potential for shared savings or gain-sharing arrangements, where the provider and client share in the cost savings or value generated by the project. This can provide additional incentive for the provider to deliver results and can align the interests of both parties towards the same goal.
Ultimately, the pricing model should reflect the actual value delivered to the client and should incentivize the provider to use productivity tools like generative AI to their fullest potential. By avoiding labor rate based pricing and adopting a value-based approach, providers can deliver cost benefits, additional value, and better profits while also improving productivity for their developers.
Conclusion
In conclusion, the introduction of generative AI models like GPT and CODEX can have a significant impact on developer productivity and ultimately lead to greater value delivery and growth opportunities for businesses. With an average of 60% productivity improvement across these roles, the overall benefits to the business would be substantial. This level of productivity gain translates into significant time and cost savings, as tasks are completed more efficiently and effectively.
However, it’s important to note that the benefits of generative AI go beyond cost savings. AI assistance can also enable developers to focus on more innovative and high-value tasks, leading to improved quality, enhanced customer experience, and faster time-to-market. To fully realize these benefits, businesses should view the introduction of AI as an opportunity for growth and value delivery, rather than solely as a cost-saving exercise.
Furthermore, it’s crucial for businesses to adopt pricing models that reflect the increased productivity gained from the use of generative AI. Traditional labor rate-based pricing models can actually disincentivize providers from delivering greater productivity gains, as their profits are tied to the number of hours worked. Instead, we recommend pricing models that align incentives for both the client and the provider, such as value-based pricing or performance-based pricing.
The use of generative AI represents a significant opportunity for businesses to improve developer productivity, drive growth, and deliver greater value to customers. By adopting the right pricing models and leveraging AI assistance effectively, businesses can unlock even greater productivity gains and position themselves for success in a rapidly evolving digital landscape.
References
- DeepCode. (2021). Feedback generation for automatic bug fixing. Retrieved from https://www.deepcode.ai/
- DeepCode. (2021). DEEPCODER: LEARNING TO WRITE PROGRAMS. Retrieved from https://www.deepcode.ai/
- Emerj. (2021). Accelerating the Software Development Lifecycle with Artificial Intelligence. Retrieved from https://emerj.com/ai-sector-overviews/software-development-ai-applications-in-the-sdlc/
- DevOps.com. (2021). Driving Software Delivery Automation With AI/ML. Retrieved from https://devops.com/driving-software-delivery-automation-with-ai-ml/
- Vasilchenko, A. (2011). Value-Based Pricing For New Software Products: Strategy Insights for Developers. Retrieved from https://www.researchgate.net/publication/238594040_Value-Based_Pricing_For_New_Software_Products_Strategy_Insights_for_Developers
