GrandQuest, venture into the unknown
GrandQuest is an indie multiplayer role-playing game with a focus on turn-based combat, grinding and collaboration. You can now join the adventure when you play the beta (free forever)!
this story can also be read here: https://jonathandiaz.me/lone-heros-trial
I want to tell you a story
I have always struggled with the artistic side of me. As of young I drew a lot, taught myself the piano, guitar and singing… I’ve always had a very artistic side. But I could never see myself as an artist until I became a programmer.
I want to share with you my experience during the past months designing, engineering and launching a multiplayer role-playing game while attending highschool — The motivations I had, the challenges I encountered and the wisdom I gained about programming, art, work and life.
And it all starts with a simple text-based game
Inspiration
Unsurprisingly I started off my programming journey making text-based games: I would take in user inputs and with IF-ELSE statements create a narrative depending on the responses of the user. Ever since I was young I had a wild imagination, so I channeled my creative forces and created dozens of narratives, all based off of the world I created in my mind. This was the point at which I realized that programming could be used as a form of art; I was an artist in the making without knowing it.
As my programming skills grew, so did my world, and soon enough EpiQuest was born. EpiQuest was the first project I felt proud enough of to call my first game; a text-based adventure with many world to explore, different mini-games, advancements, characters, sounds, etc. Although very simple, it gave my younger self an idea of what I could create if I focused my attention on a single task.
Something Different Lies Ahead
Months have passed since the creation of EpiQuest, I have become proficient in the front-end ecosystem, but no matter how many JavaScript libraries I learnt, my projects still felt very incomplete.
That was until I plunged into the dark-side which allured me so long: back-end development, when my friend Tyler showed me something wonderful: Apricot Road, a multiplayer web-based role-playing game;
Apricot Road was something new, a world hidden away for a select few of people to enjoy, and I was one of them.
Tyler: One of the reasons I feel so excited about Apricot Road is that I feel I’ve found a powerful abstraction for web-based game development
I was blown away by the project; you could have multiple players in a single world in real-time you could store their data to be used again in the future, wow! How the hell was he able to do all of this? I decided I wanted to create art like Apricot Road, so I wet my toes in the sea of back-end development by learning the MERN stack.
After some weeks of tinkering, building a full-stack social media, I became confident in my skills as a full-stack programmer, orchestrating between database, server, API, client and back. I fell in love with the interesting challenges back-end development provided (in contrast to the sometimes obnoxious and repetitive issues I encountered with front-end development) and I decided to learn Ruby on Rails.
Unsurprisingly, I began learning Ruby by creating a text-based game in the command line. It again grew in narrative, characters, locations and games, and it began to look a lot like my first game. I decided to call this command-line adventure game EpiQuest64, my first Ruby game!
I continued to attempt an LLC and did a small contractor job, both for RoR.
The 36 Questions (that lead to love)
After watching a video on Arthur Aron’s 36 questions that lead to love, I had an idea…
What if there was a platform where users could be matched at random with ideal partners and be subjected to the 36 questions? Would this lead to love between strangers? It was part experiment, part artistic expression, and part programming exercise using real-time client/server networking.
I picked up socket.io and dug in. I became familiar with the basics of web sockets, server state-management, socket-authentication, peer-to-peer reactivity, designed a UI/UX, built a matching algorithm, messaging interface, game loop, handled edge cases, and released 36Questions.
As this was going on, Tyler reaches out to me with a crazy request —
Tyler: I’m doing a contract for a company right now who need a mobile app. If you did react native, I could use a hand!
Wtf? Up to this point, I had never event built a mobile app in my life… I’m in!
During the next 2 weeks I built a mobile version of 36Question using react-native (among other small projects) to familiarize myself with the technology and began work for a golf club app as a lone programmer. This was a great opportunity to learn much more that mobile development, but also software architecture, remote work, version control workflows, and long hours of coding: I saw for the first time an ugly truth of remote work: When you’re a remote worker, no one but yourself sees how much you’re working, so your productivity is based off of results. Since I was still new to the ecosystem, worked on a very slow computer, and had to deal with a lot of bugs, I would dedicate up to 10 hours per day to working, as I watched summer pass me by.
I then joined another group of talented young innovators working on a react-native project: The Everywhere Office (TEO). I worked as an intern while going to highschool; I developed a great number of the UI’s you see today (and cleared many issues for the team to advance).
During this time I learnt a lot, including customer development, team collaboration/management, startup bootstrapping, marketing, and esoteric tooling bugs, etc… Eventually I decided to leave the team and move on. My time with TEO, however, greatly influenced me as a person and professional.
At this point I had taught myself UI/UX, front-end development, reactive programming, frameworks, databases, servers, RESTful APIs, mobile development, concurrent programming, functional programming, real-time communications, ecosystems, deployment, authentication, software architecture… Was it time?
GrandQuest — Lone Hero’s Trial
I finally felt confident enough in my software development skill-set to attempt my own multiplayer experience, like my inspiration Apricot Road.
I had never consciously worked on designing a game, the times I had before were purely intuition. But now I had to think deeply about game development, which is basically the creation of an experience; No longer was I just a programmer.
I researched game themes I could go with, and took inspiration from Apricot’s exploration theme. I still did not know what I wanted to build, so I chose a direction, got my engines going and created a prototype of a top-down multiplayer explorer (Over and over again, when I found myself not knowing what direction to go in, I just tried something and built upon it). This small demo alone taught me a lot about client/server networking, such as client/server architecture, client-side prediction, lerping, state synchronization, deterministic lock-stepping, timesteps, etc.
Open-world exploration proved to be too much workload for the stage I was at, so I thought other mini-games that could use multiplayer. That’s when I looked back at ManaHaven’s combat game and thought “Why can’t I make a multiplayer version of this?”
And so it began. I began an npm project with React, Redux, socket.io, express and … That’s when I was hit by an frustrating truth.
Tyler: Manipulating DOM via jQuery can get you pretty far, and you would find the edges of what’s possible before having to use a tool like React.
It dawned on me that I was using all these technologies out of habit, not because I needed them. So I decided, for the prototype, to use plain HTML, CSS, JS and jQuery.
Ready for Combat!
For the next couple of weeks, I sketched out ideas, mockups, designs and specifications for a multiplayer turn-based combat game where players and enemies alternated between turns in real-time until one team was completely defeated. Once it was all clear in my mind, I began to code.
For the game server, I decided on Node.js / socket.io because of it’s good community support. I then created a basic socket.io server to store the state of the combat game as an in-memory object. On the front-end, I decided to use Phaser.js as my game engine and played it by ear. I bought my first assets from online, and began to code away, adding at least one feature per day, every day. I had no other option: school just took up too much time.
From GUI/canvas state, to scene rendering, to placing lines, to organization of character data, to event selections & filtering, to the game loop, to event animations, to UI/UX etc. Eventually, the prototype took shape, had less bugs, more visuals, became more robust and I felt confident and happy with what I saw; I was well on my way to create a new experience. This was a huge accomplishment, even for a small demo: I had tackled issues I had never even thought about before and successfully created the foundation for GrandQuest.
I started development of GrandQuest with a simple premise in mind: code is not an asset. The more code you have, the less agile you are.
Technical Challenges
Development on GrandQuest has been very end-to-end: I’ve worked on RESTful, authentication, art, web-design, animation, sound-design, security, game engine, debugging, forums, community, database, deployment, etc.
Here I discuss some technical challenges that I encountered that I think are worth being shared.
Challenge #1 — Learning
I built GrandQuest with many technologies I had never used before! It took me lots of experimenting, reading documentation, and rewriting code to learn them, here they are:
- Vuejs + TypeScript
- Phaser.js
- PostgreSQL
- Redis
Other technologies used where…
- Node.js
- Express.js
- AWS
- Heroku
- Netlify
- Socket.io
- Sass
Challenge #2 —Server State
Persistence challenge: My first major challenge was a way to share the state of the game across all Node files. My first option was passing the state from file to file when required, but that was extremely inelegant and rigid. The next option I thought of was to use a real-time database like RethinkDB or FirebaseFS, however, I knew that bringing a new database would introduce a new level of unnecessary complexity and cost. Instead I settled on using Node’s file system to store data on the disk. I created a directory named storage where I stored files like world.json , combat.json and users.json . To make changes to the state of the server I simply ran fs.writeFile with the new state.
Example:
// socket disconnection handler
socket.on('disconnect', () => {
// read users from state
const users = JSON.parse(fs.readFileSync(pathToUserStorage)); if (socket.userID) {
// the user is authenticated, remove from state
const newUsers = _.omit(users, socket.userID);
fs.writeFileSync(pathToUserStorage, JSON.stringify(newUsers));
}
});
This option came at the cost of expensive operations being run synchronously in order to maintain persistence. Blocking the thread in this manner must be avoided in Node’s event loop. But at this stage of the game it wasn’t a big deal.
Eventually, around the end of the game’s development, I decided to write an in-memory store inspired by Redux that could simply be required in any file, like so:
const store = require('./store');
// socket disconnection handler
socket.on('disconnect', () => {
// read users from state
const users = store.getState().users; if (socket.userID) {
// the user is authenticated, remove from state
store.update('users', () => _.omit(users, socket.userID));
}
});
(The reason I did not use Redux was because it added code. I instead wrote my own Redux in a single file occupying less than 20 lines of code, encompassing the core functionality I needed)
Game loop challenge: In order to keep the state of the world and combat rooms going, I used a game loop to advance the game while decoupling user inputs from time and state. When it comes to the world’s game loop, I created an interval for every 1 second that would update the world state and emit it to sockets along with their user’s state if authenticated.
The combat game loop is more complex. In every tick of the loop, it will go through many conditions and take actions based on them. Examples: IF the game has not started AND there is atleast one player connected THEN start the game, IF there are alive characters in the current turn AND they have all chosen an action THEN advance the turn, etc.
Challenge #3 — Socket state & authentication
In order for users to play games and navigate the world, I must have a clearly identified connection with them with a state registered for them. So this is where we start: how do we identify users, keep a persistent state for them on the server/in a game and keep the database updated?
JWT challenge: In order to identify users we must make sure that they are firstly authenticated. Since the Vue.js client is a SPA, I decided to use JWT’s for authentication, however, I had to deal with another problem that arise when using stateless tokens like JWT: “A server has nothing to say about a token except whether it was signed by itself or not, so it has no way to revoke them individually”. My way around this was to have a token column in the database for each player and assign a JWT to it when a user created a session. When users fetched a session, the token must match that same token from the database, so we know it’s the real deal.
Identification challenge: Now that users can create, request and even revoke tokens, we can authenticate sockets. The way users can authenticate their sockets by emitting their tokens to the server, which will fetch the user’s data from the database, add it to the state and then set a flag for the user id like so:
socket.on('AUTHENTICATE', token => {
// find user with token in database
const user = db.query(`SELECT * FROM users WHERE token = ${token}`);
if (!user) return;
if (user is NOT in state) { add user to state }; socket.userID = user.id;
});
Mutual exclusion challenge: Every browser tab with the game open is a socket connection being authenticated to the server. How do we stop users from joining 2 or more combat games at once? How do we stop them from healing in a shop on one tab while fighting in another? The solution turned out to be very simple, and it involves the CS concept of semaphores and mutual exclusion (mutex). In my case, when a user joined a combat game, perform a proberen operation: check for a flag in the user’s state that told me they were in combat. If there was one, I would not allow the user to join combat. If there was no flag, I would perform the verhogen operation: set a flag in the user’s state so that they could not join. When the user disconnected, I would remove this flag. What this allowed is for multiple sockets to authenticate to a single user resource on state, and only allow one of them to use this resource at a time, in combat, in a shop, etc.
// combat join handler example
socket.on('JOIN_COMBAT', () => {
const user = findUser(socket.userID);
// mutex lock proberen operation
if (user.socketLock) return;
// mutex lock verhogen operation
user.socketLock = socket.id;
[...]
});// combat action handler example
socket.on('COMBAT_ACTION', () => {
const user = findUser(socket.userID);
// proberen
if (user.socketLock !== socket.id) return; [...]
});
Disconnection challenge: Now that sockets are authenticated to users in the server state, we need to make sure we clean the resource when they all leave disconnect, because we don’t want to remove the user from state while other sockets are using it. To do this, we simply check how many sockets on the server still have the userIDflag every time a socket with a userID disconnects. If there are no more left, we remove the user from state, like so:
socket.on('disconnect', () => {
if (socket.userID) {
const authenticatedSockets = _.filter(io.sockets, s => socket.userID === s.id);
if (authenticatedSockets.length === 0) {
// there are no more sockets auth'ed to this user
// remove user from state
store.update('users', users => _.omit(users, socket.userID));
}
}
});Challenge #4 — Client-side
Socket/Store decoupling: I decided to only have the socket be accessible through the vuex store. This eliminated uncertainty and complexity by creating abstractions with vuex actions and managing the connection outside of the component’s view. This provided a simple interface to read from for everything the component might need; socket state, user state, world state, combat room state, etc.
Flexbox layouts: Flexbox made it very easy to create responsive layouts, and I ended up creating some complex layouts using it.
Animations/SFX: Animation is no joke in GrandQuest. You will find animation everywhere. I’ve found that attention to details like animations and sound effects can make a game feel 10x more fun to play. It brings the entire app to life and makes it feel like a physical space. The UI features simple CSS animations like buttons hovering or list items scaling. For more complex UI animations I use GSAP, for example, in the “Potions Shop”, users can heal their combat character’s health for gold. When they select to do so, a health bar will slide down into the screen, the “juice” inside will slide to the new health, gold coins will be shown taken away, and away will go the health bar again. Again, simple details like these make players want to buy items or upgrade their stats just to see what animation they get next and hear the sound effects.
Game loops, race conditions and more decoupling: This is perhaps the most challenge task of all and it took a very long time to get right. Since I used Phaser as a game engine I decided to separate it from the Vue component because I knew that each piece would alone be very large and complex. Phaser works as a display for the game, while Vue works as GUI/UI, and they must be programmed in separate files. I, however, needed a way to control the Phaser game at some level from my Vue component, so I binded a game controller to the component state which is initialized when the component is mounted.
Then I had to solve many many many race conditions: there were many asynchronous events occuring when one joins a combat room that could go wrong at any moment, so I again went with a game loop design to check for the condition of the state instead of tangling events together. Racing conditions on mounting are:
- Socket is connected
- Component mounts
- Socket is authenticated
- Game is initialized
- Game loaded assets
- Socket joined room
Once we have successfully passed all of these, we must be observant of the following hundreds of variables and render the UI and canvas accordingly. The Phaser game loop is ~230 lines long, and I have hosted it here for fun! The Phaser combat code is 1500 lines long.
Apart from separating the Phaser game engine code from the component, I also separated an animations manager and audio manager for the game that take specific care of animating combat scenes and providing an API for reproducing sounds across the game.
Other challenges
- Generating character coordinates by linear-graphing
- Rendering GUI selection lists and making them interactive
- Turn-locking combat matches for all players to be on the same page
- Managing hundreds of race conditions to take each other into account and orchestrating them in a single runtime
- Used SQL/PG functions and clauses to increase performance and format query results
- Debugging Vue change detection issues
- Reproducing esoteric issues in the least code necessary
—
Personal challenges and lessons
tyler:
I have the utmost respect for anyone who embarks on a project. it's really you against yourself.
and man is that a lonely, isolating, and long journey where you work your ass off and wonder whether this has any meaning at all or whether anyone will care
Everyone struggle with this. Directors, authors, artists, sculptors, musicians.
So you won't get away from it just because you stop programming. If you want to make a living making music, you're back in the well of trying to express yourself with your talents.
This is the single-most important project I have worked on so far on my own. Throughout my journey, I’ve come to realize that embarking on a project is ultimately war with yourself, and this has forced me to adapt.
I have learnt to focus all of my attentions on a single task over an over again, allowing me to eat an elephant (Build an online RPG) by eating in chunks (one specific task, every day).
I have learnt to internalize my work and ignore external factors that take away my inner calm. Make a successful game vs Make the best game I can.
I have learnt to motivate myself and keep my own mind disciplined, regardless of external conditions, to focus on meaningful work.
I have learnt to care about the user’s experience and obssess over elegance, even in the parts that they can not see.
I have learnt to prioritize tasks and recognize when I stray from course.
I have learnt to do as much as I can with very little (An old, cheap laptop and a small budget)
I have learnt to create an end-to-end product, as opposed to programming software.
I have learnt to trust myself on my work and it’s value
I have learnt that as I grow, my goals grow with me
