We let ChatGPT run our Twitter. Here’s what we learned
From dreamy portraits to political theory, AI has thoroughly invaded our feeds over the past few days.
A week after releasing it into the wild, OpenAI’s generative language model ChatGPT has been used to design trading bots, write songs, explain DeFi, exploit (and debug) blockchain smart contracts, and invent games. Someone even used it to build an entire virtual machine inside of it.
That’s an impressive range of talents — one few humans can contest with. But just how good is ChatGPT at the broad spectrum of skills it commands? We decided to find out.
Although we’re known to dabble, coding and trading aren’t exactly our domains of expertise at Hype. We do, however, boast a solid record of creating and managing social media content for leading Web3 projects. We’re pretty good at it. So naturally, we decided to put ChatGPT to the test by enlisting it to run our Twitter account for a week.
The idea was to trial ChatGPT’s ability to create content, but also put together a cohesive social media strategy for our account.
Here’s how we did and what we learned.
Getting started with strategy
Strategy is important.
That’s why the first thing we tasked ChatGPT with was to put together a week-long content plan for a Web3 creative agency.
Here’s what it came up with:
Sound advice. So far so good. In fact, we already do a weekly news recap, so one of the suggestions on the list is something we already do.
Social media managers will swiftly notice that the strategy involves a healthy blend of formats, including educational posts, work showcases, lighthearted “get-to-know-us” content, and even some engagement bait like the engaging question and the inspirational quote.
Since we wanted to run this test only for the duration of this week, we took the liberty of shuffling around ChatGPT’s schedule, tightening it up, and adding a few extra formats to the mix.
We also decided to nix the behind-the-scenes post, since we’re a remote-first agency and sharing screenshots from Google Meet is a little cringe. Other than that, we stuck pretty closely to ChatGPT’s original plan.
Creating the content
The next step was to actually get ChatGPT to create the content.
We kicked things off with the easy items on its content plan: the engaging question and the inspirational quote.
For the former, it took a couple of attempts to generate a question we found suitable for sharing with our followers. The first attempt came back with some pretty dry — and long — options, so we narrowed the parameters on the second try. It worked:
Ultimately, we picked the second question from the second screenshot.
For the inspirational quote, ChatGPT only needed one attempt.
For this post, we also tapped DALL-E for a quick collab. We asked DALL-E to generate an image of the metaverse, which we then used as a backdrop for the quote we picked (the second quote in the screengrab above). You can check out the final result further below.
The first real issue we ran into was with the intro tweet for Hype. Since ChatGPT was trained on data sourced prior to 2021, it struggled to pick up a description for our agency that felt authentic to us and what we do.
Don’t get me wrong. It still did a decent job, but couldn’t let this description slide if we were to tweet it out from our account without revealing it was AI-generated.
We also couldn’t ask ChatGPT to create a tweet detailing our previous work, because we’ve not yet launched our showcase section (I know, shame on us). But even if we had that, ChatGPT can’t yet scan through pages on command.
We ended up giving the AI a hand and crafting those two tweets ourselves as our intention was to follow its original plan as closely as possible.
We also wanted to run a few threads on popular terms and trends in Web3 and the metaverse, so we had ChatGPT create them for us.
NFTs have been a hot topic over the past couple of years, so what better way to get started than with a thread on their uses? We asked the AI to put together a list of the seven best use cases for NFTs.
Here’s what it spit back at us:
Prior to this, we tried a similar query for the future of NFTs, DeFi and DAOs, but we noticed that ChatGPT would return identical output for all three, replacing some words here and there. That was also one of the main limitations we observed: lack of specificity.
The workaround was to assign more well-defined queries, which ChatGPT seemed to handle rather well.
We made two more threads based its input, which included a listicle-style thread on why fashion brands should explore the metaverse and an explainer on how to build Web3 community from scratch:
The only missing part from the content plan was the industry news update. ChatGPT is incapable of trawling the internet for up-to-date content, so we decided to compose a weekly news recap ourselves. As we said, we already do this anyway, so we were glad to slide in with the assist. You’re welcome, ChatGPT.
In the interest of full disclosure, we also made minor tweaks to some of the copy ChatGPT served us with — mostly to fit Twitter character limits and make it sound a bit less, well, robotic. You’ll spot the differences in the next section.
It’s go time
With all of the content ready, it was time to release it for human consumption.
Here are all of the tweets we fired based on ChatGPT’s strategy and copy — arranged in chronological order:
Final thoughts
So what’s our final verdict?
ChatGPT is seriously impressive. And we had a blast playing around with it. But if push comes to shove, we wouldn’t rule out humans just yet.
As someone said on Twitter, ChatGPT can do a lot of different things — but the quality is mostly average. It’s a “5/10.” That doesn’t mean it’s not useful. It’s a good prompt for when you’re stuck in a rut, but it’s up to you to polish the output and make it stand out.
OpenAI has done an excellent job of training the model to source information, but it often lacks nuance and depth. That’s alright, because humans are good at that.
Think of ChatGPT as a yardstick for creativity. If the content you produce and the insights that come along with it are indistinguishable from the AI’s output, that’s your cue to step your game up.
Yeah, it can help with productivity and I’m sure plenty of people will continue using it in their day-to-day for a variety of reasons. But if you’re relying on it as a content generation tool, it’s too bland to cut through the noise and make people pause scrolling their feeds.
You won’t see us outsourcing our social media efforts to ChatGPT anytime soon (not unironically, at least) — and you sure as hell won’t catch us passing off its content to clients either. 👀
(PS: Shoutout to the person who figured out we were using ChatGPT before the big reveal. You were right.)
