DEFY Product Development Update: Slapping Bugs and Droning On

Hello friends of DEFY, and fellow advocates for the disruption of Future Systems’ nefarious geopolitical influences!

This blog is a fork of our primary blog, because it’s aimed at a different audience. This blog is for our fellow hackers, engineers, and people interested in going a bit deeper in joining us in learning how to build a web3 game.

I’m Brent and I act as general manservant here at DEFY Labs. I set up systems, advise on strategy, design parts of the business, help make engineering decisions and am currently focusing on ensuring that the direction we’re taking the product is in line with what the world wants. But in this blog I’ll wear three hats to bring you content under these buckets: Product Manager, Game Designer and Software Engineer.

We launched our alpha on April 14th 2022, and it’s been a hell of a ride. It’s an amazing experience, as we’ve had 83% of players who minted an NFT mask to access the game come and play it.

Anyway let’s jump into the more technical aspects of what we’re doing. For starters, I’ll give you a feel for our infrastructure. Our game servers (BE) are written in .NET Core, and we’re using React Native on the front-end (FE). We make heavy use of PubNub to mediate game state between the FE and the BE.

Woe be Heroku

We were running some services in AWS, but a lot of our services were initially deployed on Heroku… until this week. Heroku turned out to be a bit glitchy, with servers randomly dying inelegantly. This was causing a lot of grief for our player base, as it was manifesting in a bunch of weird ways. We currently suck at error handling, so instead of the game saying “hello good player, the server has disconnected”, things just… wouldn’t work. Sorry about that!

Heroku was great for shipping the application to Production quickly without the overhead of configuring swaths of interconnected services, but we knew we needed a wider range of infrastructure options in the future, and something with proven stability. Michael, our legendary CTO, and myself talked through some infrastructure options:

• We wanted to support long-running game sessions, so serverless functions like AWS Lambda weren’t the right fit
• We didn’t want the effort of managing our own server infrastructure, so EC2 wasn’t right
• We didn’t want to become experts in container management and orchestration, so unmanaged ECS and Kubernetes were out

Further to that, we were already running our API service on AppRunner (which uses Fargate under the hood), so choosing ECS with Fargate for our game servers kept things nice and consistent. We knew that AWS is restrictive with the number of Fargate instances that can be spun up, so I got in touch with AWS to get them to increase the limit from 2 to 20. They took 5 days to get it done, and I had to answer a bunch of questions so they could be sure we weren’t going to shoot ourselves in the foot.

I used to work at AWS as a Solutions Architect, and there are very good reasons why they are careful with service limits, but still… when you’re on the customer side it sucks! I kept chasing and chasing… and finally they increased the limit on Tuesday.

Michael spent 4h working on configuring the AWS infrastructure, and then migrated us off Heroku to Fargate last Tuesday. This solved the systemic outages that were common with our Heroku platform immediately. There have been some application-level glitches since then for subsets of our users, and Michael’s been focussed solely on fixing them up until today. We added some extra logging and exception handling… found the last main issue, and everything has stabilised! This is my favourite demonstration that he shared:

That last drop in API timeouts was caused by AWS AppRunner setting an arbitrary limit on the number of concurrent requests per host. Good times.

Anyway, now it’s fixed!

As part of digging into our platform stability, we have configured a full logging and tracing platform (called Seq — https://datalust.co/seq) which has helped immensely with tracking down these issues. One of the big things Seq gives us is this nifty little dashboard, which helps us track stability in realtime:

Alpha Bug Bounty

We’ve had 280 bugs reported in the alpha. A large majority of those are duplicates (thank goodness!) and we’ve fixed a reasonable percentage of the ones impacting people the most. There were some bugs reported that could have resulted in people gaming the system too, and we’re so grateful to those of you who reported them — thank you!

We’ll be paying out the first bug bounties for those who found the bugs first in the alpha over the next few weeks, most likely in conjunction with the end of Mission 2. We’ll keep you posted!

Please keep in mind that if you find anything that could affect player’s ability to rapidly accrue FCOIN, or the impacts the security of the app or our servers, please let us know! We’re paying handsome bounties to you beautiful White Hats in our audience. Web3 needs to be proven to be a stable and safe place for everyone, and every time a hack takes place is degrades the vision many of us have for a better, more open and safe Internet for the world.

Want to Know What’s Coming?

Drones

One of our moles has informed us that a division of Future Systems (those bastards) has started noticing some anomalies in the balances of their accounts. Their suspicions have resulted in them sending enforcement drones to a limited number of towers where they will be observing tower behaviours. If they detect anything… unusual going on with the towers, they will send drones to investigate nearby areas.

Items

Currently, the only item you have in game is your mask! How tragically basic! We’re going to start adding items that you can pick up to enhance your gameplay. I don’t want to say anything too detailed yet, but they will likely be augmenting your ability to recharge your energy, execute exploits to simplify the hacking of towers, or possibly even equip you with a fighting chance to evade the drones…

That’s all for now!

As mentioned in the intro, we’re planning on sharing the journey, but we’d love your input! Please let us know in the comments what you’d like more or less of and we’ll do our best to share the interesting bits of this adventure!