Adventures in AI: MetaGPT
Assembling the AI Dream Team
Dream Team 🏀
In a perfect world, every entrepreneur would have a team of all-star developers, armed and ready to author commits at a pace that would make Speedy Gonzales look like Slowpoke Rodriquez. Intelligent Agents are a step towards realizing the dream of lighting fast application development, and projects like BabyAGI and AutoGPT made quite a splash in the AI world when they were first released. BabyAGI and AutoGPT can accomplish sophisticated coding tasks through LLM-based automatic programming, and the results can be absolutely astonishing. The approach taken by most AI autonomous agent projects is limited due to issues like memory and hallucination. By creating a system of agents with specific roles, inter-agent communication, standardized operating procedures (SOPs), and validation protocols, MetaGPT achieves results that are significantly better than single autonomous agents are able to achieve on their own.
Research 🔬
On August 1, 2023, a paper emerged outlining a new approach to AI Agents. The paper, titled MetaGPT: Meta Programming for Multi-Agent Collaborative Framework, describes how researchers instructed AI to emulate a software development company and assume various roles in the software development process. The research team created fully functioning Python applications by creating agents with specific roles, the ability to communicate, access a shared memory store, and validate its outputs. The research team’s results with MetaGPT were noticeably better than BabyAGI and AutoGPT.
Today I attempted to answer the question: “Is MetaGPT capable of being the all-star, butt-kicking team that developers dream of?”. The quest to find the perfect software development team is of great importance to me, as historically, I’ve struggled to find people willing and able to help with various side projects. Armed with my first day off in months, I decided to dive right into the MetaGPT research paper and associated GitHub repo.
The following video is a fast-forwarded screen capture of generating a project via MetaGPT and tweaking the output until it worked.
Analysis 📋
The results were encouraging, but unfortunately, MetaGPT isn’t quite powerful enough to be drafted to the software developer all-star team. MetaGPT did a great job creating simple, runnable Python programs, but the outputs weren’t perfect. In generated projects, the dependencies were often outdated and didn’t run on my system. A few times, upgrading said dependencies resulted in breaking changes, which required me to debug issues manually. Since I’m not a Python developer, debugging each issue was especially brutal. I hacked around with MetaGPT for a few minutes and managed to generate a Node-based snake project that could be played in a web browser, but unfortunately, most of the gameplay functionality was missing or broken, and I had to make edits by hand.
Despite being a bit rough around the edges, MetaGPT did a surprisingly excellent job of scaffolding projects using frameworks that I was unfamiliar with. By specifying that implementations should use frameworks such as Flask and Pygame, MetaGPT was able to bootstrap projects much faster than I would have been able to by myself. For simple projects, GPT-4 presumably had an enormous corpus of relevant data in its training set, and thus the resulting output was beautifully written, succinct, and mostly correct.
Presumably, many of the bugs I ran into with MetaGPT were due to the prompt I used not being specific enough. Unfortunately, though, likely due to my own inadequacies, every time I tried to increase the specificity of the prompt, I missed something new and played a game of Whac-a-Mole by fixing one bug and introducing another. Next time I return to MetaGPT (or any other autonomous agent framework), I plan to spend more time experimenting with prompt specificity in hopes that I will get better at prompting or the agents will get better at inferring my intents.
Closing Thoughts 💭
Modeling autonomous agents after software development teams is a clever strategy that yields exciting results. Autonomous agents can’t yet bootstrap entire SaaS companies, but they are shaping up to be a powerful tool for software developers. If you want to quickly spin up a new API or churn out examples for how-to articles, MetaGPT is a great companion. If you’re looking to replace a lazy junior developer, MetaGPT isn’t quite there yet, and you’ll have to wait a bit longer.
Want to Connect?
If you found the information in this tutorial useful please subscribe on Medium, follow @workingdevshero on X, and/or subscribe to the Working Dev's Hero YouTube channel.
