Purdue Study Shows AI Correctly Answers 48% of Programming Questions! Yay!

Mr. 6
Generative Geeks
Published in
2 min readMay 25, 2024

--

A recent study from Purdue University found that ChatGPT answers programming questions incorrectly 52% of the time and 77% of its responses are verbose. The high error rate has led people to believe that human programmers are still indispensable. This is totally nonsensical.

This isn’t about flipping a coin and getting a 52% failure rate, which would mean nothing.

It’s about a 48% success rate in writing a correct program WITHOUT the help of a technical person.

What does this mean?

It means freedom for non-tech people who no longer need to rely on programmers.

Programmers are expensive and hard to communicate with. Sure, they don’t have a 52% failure rate. They will complete your tasks correctly and have a 100% success rate, but that comes AFTER a lengthy negotiation, many no’s, rejections, and scheduling delays. Plus, even after the program is 100% correct, it might be 100% unsuitable for the market. Then, you need to ask the programmers to adjust and redo everything. You’ll face another wave of reluctance from human programmers.

Not to mention the mental debilitation in collaboration — interacting with programmers can often be intimidating for non-technical individuals. The technical jargon is the least of it. The stereotypes (“you are non-tech, I am tech”) and condescension can be demoralizing.

AI can solve all of that.

It has a 48% success rate, according to the study, in completing a non-tech’s request. We assume verbose answers can be accepted (note its 77% verbosity rate). And this is only AI’s first try. You can run AI to fix the program when it doesn’t work until it does. AI is improving rapidly like a fast-moving river. We would expect the success rate to increase after just a few months.

Hail the freedom of non-tech people. Release the burden from the techy world. Thanks to AI.

AI is a solution, not a problem. When we laugh at its error rate, we should subtract it to celebrate its success rate.

Imagine a world where you don’t need to beg for a programmer’s time or feel inferior in tech conversations. AI is like a friendly helper that doesn’t judge or intimidate. It’s there, learning and growing, making your life easier one task at a time. So, kick back, relax, and let AI handle the nitty-gritty. After all, in this evolving tech landscape, it’s about enjoying the journey and celebrating every step forward. Cheers to the future, made brighter with AI.

--

--

Mr. 6
Generative Geeks

Book author. AI enthusiast. Daily-writing hobbyist. Built a startup using a primitive form of generative AI to mass-generate ads in 2017. Stanford graduate.