OpenAI Codex and Future of AI-assisted programming

Every development in Science and Technology creates some speculations about job security or society. The invention of writing to groundbreaking advances in Artificial Intelligence (AI) and Machine Learning (ML) are all-time examples. The recent advances in Language Models such as GPT-3 showcase that code can be generated from a problem statement. GitHub (a Microsoft company) created a solution, namely Codex.

Today I got a chance to sneak-peak to Codex through the OpenAI Codex challenge [1]. The experience was quite enjoyable (I attempted only two questions in between chain of meetings ). I decided to dig deeper into the leaderboard after the challenge.

The challenge was open from 10.00 AM PST to 1.00 PM PST. Five questions (mostly the nature of the questions were tech company interview questions. Yes! you heard it! Like HackerRanek, LeetCode…). For each question, we get ten code assist attempts to solve. You are free to ignore the Codex assist too! I am sharing some insights from the leaderboard data.

Users took 23 minutes to 90 minutes to complete the competition (assumes all the 100 participants were fully engaged in this).

The outstanding performers were the five users who did not leverage the Codex and finished the challenge in 71 to 81 minutes. If you get a company interview link from HackerRank, you may get almost a similar time for five questions. If we compared the performance of a coder with eight Codex assist and finished the challenge in 24 minutes, this number can say something. An experienced developer may shorten their development time by 1/3 with an AI assistant (an early conclusion from limited data and considering Codex is not biased with bugs in the training data!).

A total of 23 participants were used five or no code assist and tool 36 to 89 minutes to solve the five questions. Understanding the relevant skill and experience attributes of these participants will be helpful in many enterprise decisions, such as quota for developers in Codex API usage! That is, 23% of the participants with experience in programming can solve the challenge with less support.

Practically such a solution should reduce the StackOverflow/Google/Bing time and distractions from the activities. The advertising punch line of “Pair Programming with AI” is relevant in this case.

The talent and data-centric logic writing are still winning though we are talking about AI to support it. So programmers are not at risk of losing against AI. As a person understanding the challenges, complexities, and errors such a Language Model can commit still it is a good start. A new culture in software development may kick start with the wide adoption of such systems.

As writing software systems become, more innovative unit testing should also excel. The test cases in the competitions were very good; if we attempted only with the AI suggestions and moved forward much of thought, you will not submit!

Codex’s success will depend on reducing the bias from its training data (GitHub) and end-user training. A programmer without AI (not Data Scientist and other series of such titles) will be replaced by a programmer with AI in the future! The platform is already under scanner from software license experts relating to GPL code as training data.

Summary of Smart Programmers and AI-Enabled Programmers

Data available at https://gist.github.com/jaganadhg/2ec0ee26de1c8c3e57dd649edb2960ec

[1] https://challenge.openai.com/ — Last Accessed on 08/12/2021