That time when the metadata update went wrong…
What can go wrong with NFTs and how to handle the fallout.
Last week we had some good intentions with our Gorecats project on Solana. We saw that a few of our holders had errors on their images — the original metadata said that the images were meant to be signed by the artist, and instead they were unsigned. It was around 6 images that had to be updated by changing the on-chain metadata to point to a new URI, and adjusting the off-chain meta data to point to a new image. We also wanted to add category information to the whole collection to make it easier for buyers.
On Solana there’s a useful script from the Thugbirdz guys which does the job for you via the command line. Tic-tac-toe, and your update is done. Just make sure that the new JSON metadata files are correct, and so forth. Sounds simple?
It was Friday night and we’d just had our weekly AMA, and the dev team was doing the update during the AMA. Sure — we told the community, the metadata update is being done, and it will all be sorted out in a few hours. This is good news, because the update had take so long to get sorted out, as a relatively low priority task in a complex project and a very busy team that is committed to to building real-value, and not speculative value…
The metadata update that we were running went wrong: The Solana RPC node that we were using kept skipping transactions (due to the load increase with the Opensea integration) and the artwork and JSONs had suddenly gone out of sync.
On 3am Saturday morning, I woke up to check the Discord — peoples cats are different from the ones that they bought. What’s going on? What happened with the metadata update? The skill here is to panic for exactly one minute, vent and then get back into acting — methodically trying to solve the problem.
Call the devs…(there was no answer).
Check for duplication errors on Magic Eden…(Oh crap…there are too many errors…to count…)
Tell the community not to panic, not to list, and not to buy…(Thanks to the discord mods…)
Call the devs partners in a different timezone…(can you call so and so about the metadata).
Check the metadata files for corruption…(looks ok, everything is in correspondence.)
Call the devs…(hey — what happened?)
Let’s run the update again…(still no change…?)
Here’s the cache file…(check the file for an hour, looks like it’s corrupted…!@#@ what happened?)
People are listing their corrupted cats…(a couple of sales went through…)
Get some sleep, let’s try again when people wake up in their different timezones…
We’re fixing it ok — it might just take a while…check on Solscan…Magic Eden’s cache takes a while to update…
Ok, it looks like it’s mostly clear.
There’s a couple odds and ends, but everything seems fine now.
Make another announcement on Discord, sort out the refunds for those that bought cats…manage the reaction…
Let’s check it all again…
It’s easier to break something than to build it. And, as simple as it seems, there are a lot of moving parts in an NFT collection. At this point, now that things are mostly back to where they were before, and the intended upgrades have kicked in — it’s been an interesting lesson for us as a team. Coordination and reliability are big, big factors in business, but things also have to be fault tolerant, especially when everybody is working in a different timezone.
Our promise is that at each iteration we will get better as a team, and continue to deliver real-value to our various stakeholders: particularly, the O.G. Gorecats buyers. Thank you for sticking with us, and being patient, things do go wrong sometimes: And, they can go wrong badly and quickly, but they can also be rectified quickly and well.
This week has been about rolling the dice, Zombie cats are coming, and we’re building up to some exciting utility that will make the whole project better, and give it long term value. So, in general, while this was a snag — keep in mind that we are focussed on the long game.