A proactive approach to performance and scaling for large NFT projects.
Written by Mansa Gills
NFT drop events are exciting, and if you’re a collector, you understand how much luck comes into play when trying to get your hands on something rare. Whether it’s a highly coveted next-gen game system (you know what I’m talking about) or the release of an NBA Top Shot pack with a rare Lebron James moment, so much depends on getting to the site early enough to get your chance. But unfortunately, factors that are outside of your control can prevent you from your chance at glory.
A Troubling Experience
When the PS5 came out, my son had to have one. Ok, I'm lying; I had to have one and I used my son as an excuse. The night of its release, I was signed on to two computers while my wife, being a fantastic partner, was signed into a few retail sites using her phone and laptop. I miraculously made it to the sales page on the Target site, added my PS5 to the cart, and nervously fumbled around with my credit card to confirm the purchase. After staring at several screens for god knows how long, this was it! I yelled from the other room letting my wife know that I got on while the screen was loading. Then something odd happened. I was stuck on the loading screen.
Now I didn't want to panic and figured the site was just a little slow since there was so much traffic. But after about five minutes, I was freaking out. By the time I mustered up the courage to press click refresh, the screen had turned white, followed by a sad emoji informing me that the site was down. Before I threw my laptop out the window like a crazy person, I refreshed the page only to find out that my cart was empty and the PS5 was sold out. It was also 1 AM, and I was angry and exhausted. In the subsequent days, I learned that I wasn't alone. Performance issues ravaged all retail sites carrying the PS5. While many were fortunate enough to secure one, there was a large contingent who shared my same fate and walked away frustrated and empty-handed.
Target, Best Buy, Gamestop, and Walmart were unprepared to handle thousands of consumers flooding their sites. Unfortunately, NFT drops have been marred by similar issues.
The Problem with Drops
For those who may not be familiar, an NFT drop is releasing a non-fungible token project with an exact date, time, and the minting price of an NFT. As interest in NFTs has skyrocketed, thousands of users overwhelm websites and crash marketplaces. This issue has one common factor–the websites are consistently unprepared for the traffic surges caused by the hype around NFTs drop. In fact, performance issues have become so commonly associated with drops that users almost expect it, like it’s just the cost of doing business. Here at GigLabs, we don’t want unfair, chaotic, and problematic releases to be the norm for our partners releasing NFTs on our platform. Their customers deserve a pain-free experience, so that’s why we decided to proactively update our platform to scale with our partners.
Performance Issues
We reviewed several metrics, including the number of API requests sent and uncached responses, uncached response rates, max response times, and database CPU usage rates. As a result, we found the following areas for improvement:
When too many users clicked “Buy” simultaneously, the site became unresponsive
We couldn’t get to the queue outflow rates we wanted due to the high volume of traffic during massive drops. Our max sustainable outflow was around 150 users per minute. Anything more than that, we noticed latency in the system and sites becoming unresponsive.
High P2P traffic slowed platform performance
When the P2P marketplace is active following a drop, thousands of users rush to the marketplace to list their NFTs, which causes the database to hit maximum CPU.
These aren’t the issues we want our clients to deal with. Performance lapses on NFT marketplace platforms, especially during drops, diminish the consumer experience. Because we take pride in the reliability of NFT Bridge, we were determined to find the root causes that typically hamper the user experience across the NFT landscape.
3 Root Causes
I don’t want to bore you with too many technical details, but I’ll give you the highlights of a few root causes for these issues:
- Frequently used SQL queries executing slowly.
- The response cache was not as effective as expected
- MySQL connection management was taking up a non-trivial amount of database CPU
Adjustments
Once we determined the root causes, we developed a plan to address these issues to provide the best user experience possible.
Database Queries
The first issue we tackled was addressing the speed of our database queries. While looking into slower-performing queries, we found many queries using an index. Unfortunately, MySQL can only choose one index to apply to a query even if there are more indexes that would cover other fields in the same statement.. To resolve this, we transitioned to compound indexes for those queries taking care to order the fields in the index as efficiently as possible. This helped significantly because we could use the same compound index for multiple queries. For example, if we used a compound query searching for org_id and token, we could use this same compound index for a query searching for just org_id. As a result, fewer rows needed to be scanned for more complex queries, enabling our database to execute queries faster.
Response Caching
We learned about a small time window when the cache wasn’t performing optimally. This lack of performance was particularly impactful during high traffic, causing more database queries than necessary. On top of this, we hadn’t implemented the compounding indexes yet, and these additional queries were running slowly. To remedy this, we updated our response caching method. Previously, when our servers processed multiple requests simultaneously and tried to check if there was anything in the cache, it resulted in excess database queries.
To remedy this, we used a distributed lock method to let one of those requests proceed while the others waited until the request finished. This new method ensures that for any given request, there’s never more than one actively running database query for the request.
MySQL Connections
Lastly, we addressed the MySQL connection management issue by implementing an RDS proxy. This method offloads the database connection management from the database to the proxy managing the database for us freeing up CPU time.
Then & Now
So what does this all mean? The short answer is that we significantly improved our platform performance. However, I’ve included performance tests results below if you want to learn a little more about how this all works. Note: We ran the following test based on a simulation of 6000 users continually browsing a site for over 30 minutes.
Wrap up
So there you have it. We set out to make the necessary performance enhancements. Hence, when building hype around your next drop, you have one less thing to worry about. We increased our sustainable outflow from 150 users per minute to 1000+. Lastly, we added a proxy to our database connection management to free up CPU time and maintain strong performance during high P2P traffic.
We feel strongly about our product, and we don’t want the common issues associated with NFTs to be status quo. We’ll continue to be proactive in our pursuit to make NFT Bridge the most frictionless NFT platform out there. So next time you jump in a queue for the next significant drop, hopefully, it’s running on our platform, and the only thing standing between you and that rare moment is the luck of the draw. Happy Minting!
Mansa Gills is a Sr. Technical Writer at GigLabs, a passionate writer and technology enthusiast who loves simplifying complex concepts. GigLabs is an Atlanta-based technology company that provides white labeled end-to-end API technology stack for brands to launch NFT storefronts and marketplaces. The company partners with brands, agencies and creators to provide easy-to-use tools for generating, minting, and selling NFTs that can be presented within fully customizable brand experiences.
