I have one month to make an MMO: Day 10

I’m struggling to keep up with the planned work, I think the extra time spent on the Twine tool has knocked out my plans for this sprint, and I’m consciously avoiding overworking, so I can’t put in extra time to catch up.

Twine/Harlowe Interpreter in Python

One of the complexities of having a dialogue system with scripting capabilities is the script needs to be evaluated at runtime, and can’t be pre-processed. This is necessary because, for example, the dialogue may need to be different depending on what the player has in their inventory. So the dialogue’s scripting needs to be evaluated at the time the player interacts with the NPC.

Since I’m not quite that insane, I’m taking a significant shortcut in making this interpreter. Something I discovered while working on the visual novel project: Twine/Harlowe at some point needs to generate some sort of Parse Tree or Abstract Syntax Tree for its own internal use, so has a function for it that we can hijack. (Harlowe is one of the languages that Twine supports)

A Parse Tree/AST is sort of a nicely structured tree-representation of a program. It’s free from any specific details like syntax, kind of like a “distilled” program. A parser can follow the tree and run the program without having to worry about things like reading individual characters.

The bit that actually generates this Parse Tree/AST is called the “lexer”. And I can save a lot of time and complexity by just directly tapping into Twine/Harlowe’s lexer rather than writing my own.

Part of the code that does the lexing

As can be seen in the above screenshot, I’m running a harlowe.lex() function. This function is actually imported directly from a copy of Harlowe which I’ve copied into the project folder. It’s literally calling a function that is normally part of Twine, and I’m having it do my bidding.

As a result, the full process to get from Twine to data in the database now looks a bit like this:

1. Edit the Twine passages using the online tool
2. Twine auto-saves to Firestore
3. When I’m ready to deploy the changes, I run the bootstrap scripts
4. The bootstrap script fetches all the data out of Firestore, and runs Harlowe’s lexer function on that data, producing a JSON PT/AST
5. Bootstrap script pushes this into Redis

Now, when a player interacts with an NPC, it looks a bit like this:

1. Player makes an RPC request to the backend with the ID of the NPC
2. Server can validate this request if necessary (e.g. checking that the player and NPC are close enough next to each other)
3. Server fetches the dialogue IDs from Redis
4. Server parses the dialogue (this is the bit I’m working on)
5. Server sends the rendered text and choices to the client to be displayed
6. Player can make make multiple-choice selections, which will cause another RPC request to be sent and a another dialogue fetch cycle

The reason I call it a Parse Tree or an Abstract Syntax Tree is because whatever it is that Twine/Harlowe produces is sort of half-way between. It still has stuff like whitespace and commas, and some structures are flat; but it’s not just a list of tokens either.

Here’s an example of the kind of structure that is produced. I’m mapping all of the nodes in the tree to Python objects that I’m defining.

The end result, once this is commanded to parse itself, should be a set of arrays that can feed directly into a dialogue engine in the client. I’ll be using Friendly Cosmonaut’s dialogue engine for this, so the parser is going to be tailored specifically to generate the artefacts that the dialogue engine wants, in the format that it wants it.

Sidebar: preaching about Git

I don’t have any other screenshots to share at this point, but I do want to go off on a complete tangent and talk about Git. It’s come to my attention that some of you are still not using source control, and this is very upsetting to me.

I’ve talked before about the importance of source-control. When someone posts on the discord I hang out in asking if there was any way to salvage a project that was lost due to data corruption or accidental deletion, sometimes my only response is:

The best way to recover your files is to hop into a time-machine, go back in time, and persuade your past self to start using source-control.

Harsh as that may be, I see enough horror stories about lost work that could have been prevented with source-control, or even just some form of backup, that it should be assumed data loss is inevitable. It’s a case of when data loss happens, rather than if it happens.

For those unfamiliar, let me clear up some terminology:

• git: git is the popular version-control/source-control system (VC/SC, these two terms are used interchangeably) that was originally created to track the development of the Linux kernel. This is the name of the technology/software that is used to keep a track of changes in code, but it’s not the specific client or host that is used. Alternative version-control technologies include subversion (SVN), mercurial, and perforce.
• git host: git is known as a distributed version-control system, meaning it doesn’t really need any kind of “git server” to work, you can in fact use git directly between two computers, or perhaps more unusual, between two folders on the same computer. However, for most programmers and teams, a git host is used. The git host is usually a server that is treated as a centralised git repo; and usually adds a few usability and collaboration features like branch control/restrictions, and pull requests. GitHub is a popular git host, along with GitLab and BitBucket; all services have some form of free tier. Using a git host means git can now do double-duty of keeping track of code changes as well as acting as a sort of backup, since a copy of your code is kept safely on another server.
• git client: git is/was actually the name of the original program that did version-control. However, since then many different clients have been written that provide other user interfaces or are ported to other operating systems, so you don’t have to use the original command-line git program if you don’t want to. Graphical user interfaces for git have certain benefits and disadvantages. Examples of git clients include git the original command-line client; something people refer to as git bash which is actually not its real name, it’s just git in a MinGW environment running a bash shell for Windows; GitKraken; GitHub desktop, the official client of GitHub; SourceTree. There are also git-integrations where git is built into IDEs and programmers’ text editors. I use Atom for a text editor, it has a nice git integration. GameMaker Studio 2 also has built-in git, but for various reasons I don’t recommend using it.

So, if talking about my own source-control setup, git is the source-control system I use, GitHub is the git host I use to act as a centralised place to keep my git repository, and for a client, I use a mix of git bash, and the integrated git inside Atom. I’m also now using GitKraken for this project for a reason I’ll describe below.

Git command line vs Git GUI

Programmers often like to have this discussion about whether using git from the command-line is superior to using a GUI. There’s often gate-keeping to the effect of “you’re not a real programmer if you use a git GUI”, which is non-sense.

Example of git in the command line

Git can be used from either command line or GUI. For the majority of tasks, I’d say about 99% of the time, a GUI is fine to use, and probably faster and easier for most people (of course, if you’re a command-line wizard, more power to you). There is unfortunately a 1% of the time where you need to do some gnarly things to your git repository because unfortunately git isn’t the most user-friendly system and is doing quite a complex task that has a high chance of going awry. In those moments, you probably need to use the command-line to fix it (although it has to be said that copying the recent changes out; downloading a fresh copy of the repo from the origin/host, and copying the changes back in is a viable if hacky workaround to most issues).

Personally, I use the command-line most of the time if I don’t already have something open, since it’s fast to launch. But if I’m working in Atom for example, I’ll just use the built-in GUI tools there; and also if I need to inspect changes and select the files I want to commit, I’ll do that through Atom since it has a nice diff viewer, and it’s faster to click on the files rather than type in long paths in a command-line (even with tab-complete).

Example of Atom’s git sidebar. Most git things can be done with a few clicks very rapidly.

Fact is, you don’t have to be either one or the other. You can pick and choose, switch as the situation requires it. I’d say for beginners it’s easier to just use a GUI, but on the long run, it’s worth learning about the command-line.

GitKraken

As mentioned, I typically use git from the command line or Atom, but for this project, since it’s split over now five repositories (backend, game client, website, maps, twine tool) which I sometimes have to jump between to implement a feature, I’ve found it useful to use a different git client that can keep all the projects open for quick switching between them. I already had GitKraken installed from before, so I decided to start using it.

GitKraken with all four repos open. Notice the lack of branching, I’m a bit more lax on that when working solo, and at early stages in development.

I think GitKraken is probably the nicest-looking git client out there, and it has a free tier, as do most git hosts. So you have no excuse not to be using some form of source-control for your projects.

Day 10 Task Summary

By my estimation, I’m around one day behind at this point. I should have finished the dialog engine in it’s entirety by now, but it’s looking like it’ll take me another day to do it. This may be the first sprint that my plans are lapsed slightly.

That’s all for today, tomorrow I’ll hopefully finish the dialog engine, and make a start on other tasks.