The Technology Stack powering Lost Relics

A technical deep dive into how I used Enjin, .Net Core, PlayFab, Azure and Unity to build the first Enjin Mainnet ARPG game

Published in

Forgotten Artifacts (now known as Lost Relics)

18 min readJan 5, 2020

It’s been 8 months since I released the first playable version of Lost Relics (May 13th 2019 for Founders and May 23rd 2019 for everyone else) which ran on Enjin Ethereum Mainnet.

Lost Relics was the first platform to join the Enjin Spark program. I missed out on the Early Access program because I was busy with other work but I jumped at the chance to join the Enjin ecosystem and start my adventures in the blockchain space.

Initial Release

The first release was quickly followed by the first event called ‘Stampede’ and I used it to see how well the systems I’d built performed under stress and if there was anything further I needed to build and maintain. The Stampede Event concluded around June 13th 2019.

I had originally planned to shut the server down after the first event so that I could spend some time reorganising, planning and improving. However everyone who had played, enjoyed the event so much that they asked that I keep the server running so that they could continue playing.

With only some minimal downtime for updates, the game has been live non-stop now for 8 months and counting!

The second event ‘The Fallen Tablets’ started on July 1st 2019 and tested passing Enjin Blockchain assets between players to complete a goal.

As a reward, players were given the chance to find high tier loot within a secret treasure room, after activating and passing along one of the 3 tablets.

I launched using PlayFab for my backend services, which allowed me to rapidly get up and running with crucial systems such as Player accounts, Inventory management and Server side scripting.

PlayFab is an amazing service if you’re just starting out. They handle a lot of systems for you and have some fantastic features such as Player Management, Player Inventories, Item database, Analytics Dashboards and Leaderboards. I was frequently surprised when I went to use something at how simple and easy it was, which also meant I didn’t need to write any code for that feature myself.

I highly recommend PlayFab, however your mileage may vary depending on what you’re attempting to achieve.

I ran into issues with the way I needed things to work and I detail some of those challenges below.

In the 8 months since I first used them they’ve done some significant upgrades and changes and some of the issues I ran into may have been addressed.

Enjin Integration

Initial Integration

I publicly announced Enjin integration with a full user path from login to obtaining an item on March 29 2019.

While it’s incredibly quick to show that you can do this process by using Enjin’s Unity SDK you definitely don’t want to release your game by using the SDK to do write operations. Things like creating, sending, minting items, etc.

Client side SDK GUI

The SDK includes 2 main components. An editing GUI for managing your platforms, and an API for querying and mutating. The GUI makes it very simple to create, edit, mint and melt your items.

When I first started, this was my saving grace because it allowed me to visually see what was available and what I could do, without directly querying GraphQL, which I wasn’t yet sure how to use.

I could see and edit my items all in the one place.

Eventually you will want to move this process to something automated though. While it’s a great introduction, it’s not something you should use long term if you’re creating many items.

Recently Enjin have updated their web console to include the above editing tools and it looks like you can do all the same straight from their website now too which is awesome.

Client side SDK API

The API code is a wrapper around the GraphQL calls and includes code for connecting to the pusher service for updates when things like the wallet balance is updated.

This API should only ever be used with an Identity that’s had permission limited to only views and any permissions that give the ability to modify should be removed. The only view permission you should need is viewBalances.

You then enter your login details into the SDK and it’ll save them for use from your client.

Be aware that these details are stored in plain text in your game binary. If you do this, you give whoever downloads your game, full access to that Enjin account so make sure it’s locked down.

You could encrypt these details in your game binary, but it’s actually much easier than you realise to reverse those details and recover them from your game.

Personally I’d warn against doing this because you may accidentally add your admin details while developing, forget to remove them and release full access in a build.

You’re also going to be doing more queries to Enjin because your server still needs to query the same information to validate operations and the client then needs to query them again. If Enjin implement quotas for api calls then you may need to find creative ways later to reduce your client calls.

If everything is on the server instead, you can add caching there and just return a snapshot to the client. If you store that cache on your server and Enjin or Ethereum goes down (This has happened twice in the 8 months I’ve been live), your players will still be able to use their items.

I DO NOT recommend using the Enjin Unity SDK (or any other client engine SDK) for anything important such as creating, minting, etc for your game (I.e. if you send an item to a player for completing a level). You want a server that you fully control to be the authoritive agent and the only entity that does these operations.

In my case, I have two inventories for a player. A virtual inventory (Any items that aren’t blockchain items, or items that are blockchain items and are sitting in the queue waiting to be sent) and the blockchain inventory.

The server queries both, combines them, caches it and sends it to the client. The cache is flushed every now and then but the client also monitors the pusher channel for changes and tells the server about this during the next update and if changed, the server flushes the cache for that user too.

Enjin’s Api is GraphQL

At it’s core the Unity SDK calls Enjin’s GraphQL service.

I used Enjin’s Unity SDK as a learning resource. The majority of my early knowledge about how things should be put together and used, came from reading the source code for Enjin’s Unity SDK.

If you’re waiting for other SDKs such as an Unreal SDK, I recommend you instead just dive in and get started by directly calling GraphQL from your server, since that’s what you’re going to eventually need to do anyway.

Enjin Recommendations

Use a server

Ensure all of your blockchain mutation operations take place only on a secured server.

You never want your client (be that native Windows, Mac, Android, iOS, Web) to be executing GraphQL calls directly because it allows anyone with enough knowledge to do those calls themselves outside of your client/game.

Queue your items and send them out in batches

When I released the Stampede event, I would send items to players as soon as they successfully exited the Dungeon.

This worked fine and I sent thousands of items out very rapidly.

That is until the Ethereum network got congested.

I discovered that you can have about 16 pending transactions on your account (waiting to be mined) before any of Enjin’s blockchain related calls would start returning an error. (I.e. Send, Mint, Create, etc)

If you didn’t track which items you were attempting to manipulate you now have no record to fall back on.

Items lost during this time in Lost Relics are understood to have been lost in the lava of the dungeons.

It also costs gas for each transaction and the amount of gas you pay increases with the amount of data.

Back then I was sending a single item to a target wallet in a single transaction using a standard send. I spent a lot of ETH back then just sending out items 😅

Instead, I recommend using Advanced send for all your sending purposes. You can send multiple items to multiple addresses, all in the one transaction.

There are limits because the size of the transaction itself has a limit but a rough guide is no more than 100 entries in the transfer array (The count of the items doesn’t matter if you’re sending FTs, but NFTs can only be 1 per entry).

Store the returned transactionId

Almost every mutation call to Enjin’s GraphQL returns a transactionId. It is important to store this because you can use it to query the state of what you just did. (Don’t just store it in a log, but store it with your data so you can use it to query state later)

For example if you issue a send, you can use this transactionId to determine if the send was successful, still pending on the blockchain, or if it failed.

A common graphQL query I use when investigating a transaction is:

query {
  EnjinTransactions(
    id: TRANSACTION_ID_HERE,
  ) {
    id
    transactionId
    type
    state    
    error
    nonce
    token {
      id
      name
    }
    retryState
  }
}

This query returns enough information in the one call to determine the cause of an error or the state if there was no error.

Most of my server processes use the above query in order to determine state.

For example to determine if someone successfully completed a Quest, I use this query to determine if the transactionId that I issued to the player for the requirements for a Quest were successfully sent to the game wallet.

I describe more about the Send Queue under the “Enjin Blockchain Send Queue” section.

Advanced send is awesome

You can use Advanced send to send items from yourself to multiple people. You can also use Advanced send to request items from people. I used this to implement the Quests you see in Lost Relics.

In the example here, I’m sending 20,000 FTs of TOKENID1 and 10,000 FTs of TOKENID2 from myself to 2 different wallets in the one transaction.

mutation advancedSend {
  CreateEnjinRequest(identity_id: SENDER_ID, type: ADVANCED_SEND, advanced_send_token_data: {
  transfers: [
  {from_id: SENDER_ID, to: "0xTARGETWALLETADDRESS1", token_id: "TOKENID1", value: "20000"},
{from_id: SENDER_ID, to: "0xTARGETWALLETADDRESS2", token_id: "TOKENID2", value: "10000"}
  ]})
 {
    id
    encoded_data
  }
}

You can add more items into the transfers array to transfer more at once. Just make sure that you don’t add more than about 100.

To request items, just use the identity_id of the user you’re requesting from.

The identity_id of a user is specific to your platform and they must have linked their wallet and approved ENJ spend

Ensure you allow for Blockchain Congestion

When you’re testing on Kovan, transactions occur very quickly. Most of the time this is also true on Mainnet, however sometimes there are issues.

Congestion and inreased gas prices is something you should plan for.

Common issues I’ve encountered are:

Ethereum is congested and the gas price has risen so that transactions you or your player have signed are no longer being picked up by the miners (because they were signed with the lower gas fee)
Bugs within Enjin’s systems have resulted in transactions not updating within their GraphQL system, even though they’ve successfully processed on the blockchain

Either design your blockchain interactions so that they’re less in the moment or ensure you have a way to manually override processing in the event that the processing time is exceeding what’s acceptable.

You will definitely want to ensure any blockchain interactions can be interrupted and resumed later in the event that gas prices have caused delays.

For example if you allow your players to craft an item, ensure that the crafting process gives enough information of the progress and then allow the player to interrupt the process and resume later if they deem they’ve waited too long.

You can see how this works in Lost Relics by completing a Quest or opening a Storage Trunk.

In addition, ensure you’ve designed your server logic to allow you to override the state checking of a blockchain transaction.

In my Admin panel I provide a button that allows me to force the blockchain state to successful. I’ve used this quite often over the last couple of months when the blockchain transaction correctly processed, but wasn’t being seen by Enjin. (Just make sure you don’t approve transactions that haven’t successfully processed, which you can do by checking the transaction hash on EnjinX)

Providing this fallback has allowed me to manually verify the transaction was successful and then push the Quest into the ‘Ready to claim reward’ state so that my players can continue.

Backend v1 — PlayFab

I initially released with my backend powered by PlayFab. I used the item catalog and drop table features quite extensively. In a traditional game where there are unlimited items the system works extremely well.

Limited supply items

If you’re managing limited supply items however, you may run into some of the same issues I did.

As the majority of my items were limited in supply, I needed to accurately ensure that I wasn’t giving away more items than I actually had in the wallet. It would be bad to show a player they’d just won the Godslayer but not be able to send it to them because I’d run out of stock.

PlayFab has a ‘Limited Edition’ feature which you can enable per item. This appeared to be exactly what I needed to use, however they have an arbitrary limit of 100 total items. (I believe this may have been raised to 1000 now, but that number is still not sufficient if you have any number of items above that value)

This was the first hurdle I encountered and so I had to come up with a creative solution around it. To do this, I tracked seperate custom data with the item. This custom data tracked the Total supply and Remaining supply.

Every time I gave an item to a player, I’d check if this value had dropped to 25 and if it had, I’d top it up to 100 (if there was sufficient in the Remaining Supply). It was janky but it worked and allowed me to continue working using their systems.

Server Script

I personally prefer C#. I started my early coding days in high school, learning QBasic and later Visual Basic, then moved to C and C++ when I started working in the Video Games industry. I picked up a bunch of other languages along the way, including those used for the web (php, javascript, etc).

PlayFab uses Javascript for it’s server side scripting and while I know how to code in Javascript, I prefer C#. The entire rest of my stack was all C#, from Unity to my custom Tools. So for me, this was a downside but something I could live with for the moment.

PlayFab have made inroads to allow you to execute C# and I believe they’re close to releasing this soon (it’s currently in private preview)

The first Server Side code written in Javascript

A limitation of PlayFab’s server scripting is that your script can’t take more than 10 seconds to execute. If it does, it’s terminated. (In my experience it’s terminated after about 9.3 seconds).

While the majority of use cases will never come close to this limit, I started hitting it within about a month as I was adding more and more Enjin API calls.

Enjin’s GraphQL calls from PlayFab’s servers were sometimes taking up to a second to execute depending on the amount of data I was querying (for example the amount of items players had in their wallets), or if they did a new release on their server. Mix a couple of Enjin calls with PlayFab calls and it quickly adds up.

If there were any additional networking hiccups a single Enjin call could take a couple of seconds or longer.

As I added support for more multiverse items and more calls to Enjin, I discovered that functions written in PlayFab’s server scripting were only partially executing.

I had no idea how much of the function had run and discovered the only way to increase Execution time was to upgrade to the Enterprise tier.

I had already upgraded from Free to Indie and then to Pro in order to get some level of Support package a few weeks before and I wasn’t prepared to upgrade to the highest possible tier just for this one feature

Backend v1.5 — Temporary Bandaid

There was no way around the execution time limit and so I had to find a solution quickly.

I decided to convert all of the server logic and other code from PlayFab server scripts (javascript) to C# code and run it on Microsoft Azure as a Web Service instead.

This allowed me full control over how long scripts ran and meant I could still use PlayFab’s services from my server.

It took about a week for me to convert all of the code over. I was now running my code on my own Azure web service and relying on PlayFab for all user accounts, items, inventory, leaderboard, etc.

Backend v1.5 Sunset — Goodbye PlayFab

Over the course of my time on the PlayFab platform I encountered several bugs, including a critical bug that affected the Limited Supply features I was using. Each of the issues I encountered I had no control over and had to sit and wait for PlayFab’s support team to respond.

The Free and Indie tier offered only Forum help. Response time was usually 24–36 hours.

For some of the more critical issues I encountered I ended up upgrading to Pro because it offered dedicated support.

Unfortunately dedicated support took almost as long as the forums to get a response (comically I got a response on the forums from their free support faster than dedicated support) and the dedicated support officer (before resolving my issue) asked if they could close the issue before addressing a solution.

It seemed like they were super eager to close issues, rather than resolve them 😫

Additionally I’d had a long standing issue where once there were over 2,000 item log events on a player account, the account no longer loaded.

I suggested they could simply add paging support to fix this.

This issue was never addressed and PlayFab support deemed it a ‘Future Feature Request’ that needed to be voted on.

As more of my players had items added and removed to their accounts, more accounts were no longer loading.

The support issues and this long standing issue were the final straw for me.

It was time for a fresh start using my own custom solution.

Backend v2 — Custom Solution

I’ve written many backend services over the years so I knew I had the ability to take on this mammoth task of recreating a large majority of PlayFab services.

It was now sometime around mid August 2019 and I started work on removing the reliance on PlayFab. I recreated various systems one at a time and brought them online once I was happy they were working similar to the PlayFab offerings.

I ended up continuing with Microsoft Azure as I’d had the most experience with it in the last couple of years and they natively supported the C# .Net Core stack I was already using. I created a database powered by Azure Sql and I made use of their CDN, Service Bus, Blob Storage, and Application Insights.

First I created my own User Authentication system which relied on passwordless email authentication (similar to how ‘Forgot Password’ mechanics work). Once this system was transparently handling the Authentication side I started work on recreating the Item database.

Once the Item database was working smoothly I added player inventory support, player account management, adventure tracking, leaderboards, shop and an administration backend among other smaller services.

On September 17th 2019 I successfully launched the new backend with a complete refresh of the website as well.

After a few minor fixes it’s been running fantastic ever since as I continue to build upon and improve it.

Administration Panel

Some developers will simply setup direct access to their databases, run SQL queries to read and write data and manage their accounts and that’s enough for them.

I prefer to reduce the human error component and create interfaces that anyone can use.

I created a simple Adminstration Panel to facilitate this and to give me a control panel for managing and administrating Lost Relics.

I use the Admin panel to create my game items (Including Enjin items), setup new events, adventures and quests, all without the need for executing hand made SQL commands and to ensure particular rules are followed when entering the data.

Editing an item

An item Pending Blockchain creation

If the system detects that an item is a blockchain item but hasn’t been created yet, I can create it straight from my Admin panel.

When I click the ‘Create Blockchain Item’ button, the item is added to a Queue for processing. It’s picked up by a server Web Job which will issue calls to Enjin to Create the token with the entered values, set the metadata and mint all the instances.

I highly recommend setting up automated systems like this for your own projects as it removes the human error component.

Enjin Blockchain Send Queue

After the Stampede Event I realised I needed a queue to process my transactions at a later time.

The benefit of a Queue is you can check for several preconditions before you send your items.

I check for the following before sending items:

Is the Send Queue enabled? If it’s not, delay sending
Has the last transaction we sent, completed successfully? If not, delay sending
Is the Gas price a reasonable price? If it’s not, delay sending
Are there at least 100 unique items to send out (i.e. 1 full batch) or is the oldest added item greater than 12 hours? If not, delay sending

If all of the above preconditions are met, then I will send a batch of items out.

I store the resulting transactionId and check it regularly to see if it’s successful.

Step 4 is repeated till there are no more full batches left, or it’s been about 30 minutes with no change in the transaction state.

The entire process is run every couple of minutes as a Web Job on Azure.

Unity 3d Game Engine

Game Engine History

When I used to work in the Video Games industry, every company I worked for had their own in house game engine.

Each engine had their own pros and cons and was usually limited by the knowledge and expertise of the staff at the company.

If you changed companies, the majority of the Engine knowledge you had was now useless because the engine at the next company was so very different to what you were using before. There were usually no standards enforced across the industry and each company ventured down their own paths.

Unity has definitely changed this. It added much needed competition and finesse which drove down prices of Engines such as Unreal. The companies I worked for never licensed other Engines back then because the Engine licensing costs consumed almost one third of the budget of the game.

The power of the Unity Asset Store

I’m a programmer by profession but I also enjoy art, audio and design. The ease of use of Unity, coupled with their Asset Store has allowed me to rapidly create Lost Relics from the ground up all by myself.

This kind of ease of use and power was unheard of 10 years ago.

I curate a lot of my art assets from the Unity store and then customize them to fit within the Lost Relics universe. It’s a common process for indie developers to do this because it gives them such a powerful boost in productivity.

Unity Editor

I rely on Unity to bring the world of Lost Relics to life. If you’ve ever written an Engine you’ll know just how much Unity handles for you (Hint: It’s a LOT). I design the levels, create code for the gameplay and preview it all within the Unity Editor.

Unity does a lot of the heavy lifting so that I don’t have to, instead I can concentrate on the game and creating engaging and fun experiences for everyone to play!