‘Bubble Jump’ — Building an augmented reality game during my internship
Taking a detour — Machine learning internship
As a third-year student in Applied Informatics at UCLL, I started looking for an internship early in the year. At the start of my pursuit of a relevant internship, I quickly distilled all the options to the four most interesting ones. One of them was a project at Brainjar, a venture of Raccoons specialized in artificial intelligence.
The objective was to create an application that turns photos into paintings. During my job interview, I discovered that I slightly mistook the project for an augmented reality application while it actually was a machine learning application. Nevertheless, this project was still the most attractive option, so I decided to take on the challenge, knowing I had no prior experience with machine learning.
Luckily, my mentors were open to teaching me the ways of machine learning. I, however, quickly realized that machine learning, while very useful and interesting, is very different from the classical programming I liked and was used to. Thus, after three weeks, I went on a different path and continued my internship on a Craftworkz project — another organization within the Raccoons ecosystem. I am grateful Raccoons allowed me to work with them and later anticipated my interests by offering a different project.
Take two — Augmented reality internship
The second project was right up my alley: creating an augmented reality game. The purpose of this game was to enable a player to jump around a (physical) soda can from one platform to another. When one of these platforms is in the range of the player, it should light up. If the player then lands on them, they should react. Of course, these platforms need to be reachable from the previous block.
In addition, to make it a real game, there should be scores, lives, and power-ups. One of these power-ups should propel the player upwards, so we called it Bubble Jump. Since a game should have sound, particle effects, and even vibrations on specific actions, I had some work to do.
Initially, this project was supposed to be a one-person internship by Jannes De Koning, a Multimedia & Creative Technologies student. As I started later in the semester, Jannes had already done much research and was just about to refactor his code. This head start allowed us to go more in-depth into the project. This way, I quickly learned what the goals were, what technologies were being used, and why he made certain decisions. Because of this, I quickly learned all about the project and was able to contribute to it very quickly.
This blog will give you an overview of all the decisions made along the way, technical advances, and the result.
Week 1–2: Image tracking
Getting started
We jumped in and started by dividing the tasks according to our strengths. We decided that I would be responsible for the technical aspects and that Jannes would handle the graphical aspects of the project. We made this decision because Jannes has a graphics-oriented background while I have a technical one.
In addition, we decided to design a rough architecture and divide the project into prioritized issues. We then grouped these issues into deadlines. Since Jannes’ internship would end six weeks before mine, we split the project into two versions:
- A version without a back-end and thus no ability to store data, but with the complete gameplay implemented
- An extension with a back-end and the ability to store data
The game’s objective was to scan a physical can and use the actual can to play the game ‘on’. To do this, we decided to use image tracking, which we could quickly do with 8th Wall. We encountered one issue, though, because when you begin scanning, you cannot open the application on a PC. Moreover, the application would not run well on Jannes’ smartphone, so we decided to implement the ‘can-scan’ feature at the end of Jannes’ internship.
Bugs, bugs, bugs
Our supervisors, however, asked us to move up the deadline of this feature as it is an essential part of the game, so we needed to find a way to implement it as quickly as possible. During the second week of our collaboration, we started the implementation, which ended up being more difficult than expected because of an unforeseen effect of the platforms.
Let us explain the unexpected effect in question. Our game detects when a player touches a platform with a collision box; however, this does not scale if the platform is scaled indirectly. As the level was scaled to the image of the can, it caused the platforms to be incorrect. Eventually, we fixed this by generating the level after the image was found.
Another problem we encountered was that the material we used to hide the platforms behind the physical can only worked on elements rendered at a later stage. 8th Wall did not describe this problem in their documentation; we only figured it out after testing.
In the first part of the project, we implemented Bubble Jump, lives, and timing elements — all visible in the following video. As you can see in the video, the Bubble Jump was not ideal and only worked fully when the frame rate stayed consistent. We fixed this later on in the project.
Week 3: User interface
Up next: the graphic part of the assignment. Jannes started on the 3D model and animation of the character, but unfortunately, he fell ill for three days, which meant the deadline moved up a few days. Luckily, we set up our collaboration so that this did not impact my ability to work. I was able to implement a couple more fixes:
- Win conditions
- Functional UI (user interface)
- Fix the wiggle of the level when tracking images
As you can see in the following video, the UI is functional but not very intuitive nor pretty. However, this was not the point of this specific implementation of the UI, as we simply wanted a usable UI for testing the functions. Later on, we would completely redo the UI.
Week 4: 3D models and animations
This week was all about the player character and animations! Or at least, while Jannes was busy making assets for the game, I tried to project the player’s score on the can. However, it was not readable (as seen in the following video), so we decided that the previous UI was better. In addition, I added sound to the game and fixed some bugs.
Week 5: Finishing v1
As this was Jannes’ last week, we decided to focus on finishing version 1 and preparing for the internal presentation as a dry run for Jannes’ finals presentation. Finishing v1 entails that I prevented the player from falling through the platforms, making the game much more playable. Initially, I planned to do this the week before but ended up working on it a little longer. I also added particles to the game and connected the level generation with the physics making the Bubble Jump much more reliable. Jannes added some more assets. For example, he created an opening animation where the player walks out of the can at the start of the game. All of this you can see in the following video.
Week 6–8: Back-end creation & UI touchups
Back-end
In week six, I was once again working alone. Because we finished version 1 and most of the major gameplay features that do not depend on storing data, I started creating the backend. I chose the Express framework with a MySQL database to make the backend as I already have some experience with both these technologies. I started this by determining what the backend needed to do:
- New users should be able to be created
- A user should be able to log in and out
- A user should be able to save their high score per level mode
- A user should be able to buy and use skins
After deciding on the backend calls, I started designing the database. Once the database was created, I started implementing everything. The backend implementation posed no problems. The integration with the game, however, caused some issues.
As 8th Wall hosted the game and used an HTTPS connection, and I hosted my backend locally and used an HTTP connection, modern browsers did not allow my backend to set cookies on the game website for security reasons. This issue would not be a problem in production because the backend would also use an HTTPS connection. Thankfully, Firefox allows you to disable this security feature.
Revisiting the UI
Once I integrated the backend with the game, my supervisor asked me to focus on reworking the UI. I started by creating some wireframes to see how I wanted the UI to look. When I finished wireframing, I implemented the design. By the end of the week, I developed an option to show the player’s predicted jump path. In addition, the jump strength is now visible to the user.
You can see the improved UI in the video below. I am thrilled that I found the model-viewer web component, which allows the 3D models to be shown in the UI. This way, I can animate the model in the UI instead of only showing a still image of the skin when selecting one. The UI is also a lot more intuitive and prettier.
Week 9–10: Documentation
In the final weeks of my internship, I mostly wrote documentation and fixed some minor bugs. The result of this you can find here. Unfortunately, this is a demo project; thus, at the time of writing, you’ll only be able to use guest mode since we will not host the backend permanently.
Looking back at the internship
I am very thankful for this opportunity. So I would like to thank Raccoons once again. I found my time working for them a lot of fun. Their work environment is open but professional, and their employees are very friendly and approachable. I would recommend an internship at Raccoons.
The second project turned out even better than my expectations. The collaboration with Jannes turned out to be an absolutely positive experience for both the project and my internship. I believe the results would be completely different and inferior if either of us would have worked on this project alone.
Finally, I would like to thank you for reading this blog. I hope you enjoy playing the game, and I would love to hear about your experience and high scores.
