vj for your home; a postmortem

Yesterday I launched ‘vj for your home’, an experimental toy web app that you can use to generate trippy visuals. In this write up I will explain why I made it, what I learned along the way, and my future plans. All of the images in this article were generated by ‘vj for your home’ and shot using the built in screenshot tool!

Some history

I’ve been freelancing full time for about two years now, starting with AR development in Unity. My first big project was a training application for Hololens, and after that I mainly did Apple ARKit projects using AR Foundation by Unity. Web dev was out of the question for me (as a C# dev, ew!). But in late 2018 I got a simple job to make a virtual snow globe for a digital holidays card. At first I thought I’d just do it using Unity’s WebGL build option, but that did not work at all on mobile–by default the web export actually blocks you from running it on mobile¹.

I had to go with JavaScript. Luckily I had some tiny sliver of web dev experience from building blogs in the late 2000’s so I wasn’t completely lost, even though the web dev landscape had changed dramatically since then. I threw myself in the relative deep, stumbled my way back out of it and delivered a neat little 3D web experience. A good learning experience! Later in 2019 I made a game using Angular which was also a great learning experience, because I got to see what a proper way of working with web technology was like. Angular also introduced me to TypeScript, which I found to be a much more comfortable coding experience².

After that I got a job to make something using 8th Wall, which is a web implementation of the SLAM algorithm, which ARKit and ARCore also use. That went well, and it really solidified the idea that I need to get comfortable with web development, because much of AR is moving there. And for good reasons: most if not all of my clients would much prefer to not go through the app stores and have the application available for a wide range of devices. It’s an easier sell.

I recently had a friend over and we played some EarthBound, and I got inspired by the beautiful battle backgrounds. I decided to make a web app that sort of makes those kind of visuals. I did some vj’ing in the past using my own Unity project, so I knew what to do. But I would express it in TypeScript and Babylon.js, which I also wanted to learn. I also decided to use Sass for CSS and Pug for HTML.

How it went

It mostly went smoothly! For my previous web projects where I had to use 3D graphics I relied on three.js, which is good, but I felt that Babylon.js was an overall better experience. The API makes more sense (it’s definitely more modern), and it’s built with TypeScript from the start. Three.js might be a bit lighter, but that’s hardly a concern in 2020, or relevant for this project. In any case, it was reassuring to know that I can jam out a project like this in relative short order: 55 hours.

One thing I do really like about web development is the iteration time: transpiling from TypeScript to JavaScript is fast, so any change that I make in my code I can test out almost immediately. Unity on the other hand can take a long time after a code change (because it has to compile). The bigger the project gets, the longer it takes. These seconds add up and can get me out of the flow of working. On the other hand, I feel like I can’t completely trust the compile time guarantees for TS to JS, which means that sometimes I’d get errors during run time that were vague at best, cryptic at worst.

Sometimes Sass would also complain, because of a white space not being right or something. Easily fixed by formatting the file, but because of that I will probably use SCSS next time instead.

Other problems I ran into were getting non-TypeScript libraries into my project. Seems that without TypeScript typings it becomes a pain to consume JavaScript libraries. I spent some time trying to get that to work, ultimately deciding I might as well write the functionality I needed myself. I found it hard to find clear and concise documentation about this.

On the other hand, adding a library that supposedly does support TypeScript doesn’t always work either. I wanted to use gpu.js to get complete control over all image operations, but it just wouldn’t work. The error messages weren’t helpful either!

The bundle size (as of writing) is massive for web standards (~7mb). I believe it should be smaller; the tree shaking might not be working correctly. Not sure why. Again, clear and concise documentation is hard to find in my experience.

I am sure there are ways to get the above issues resolved. I am sure I am doing something wrong. But if JavaScript (and TypeScript to a lesser extent) is so widely used and I can’t find the solution in minutes, then there’s something wrong on the other end too in my opinion.

In the end, I do quite enjoy web development. The fact that iterating can go so fast is really good for my flow, and distribition is super easy. It’s just very frustrating when something doesn’t work and it’s hard to find out why. Of course C# has it’s pitfalls too, but I’ve been working in C# for over 5 years now so I suppose I’ve learned to recognize most if not all of the frustrating things.

UI

For UI I initially wanted to use Dear ImGui, and found a nice JS implementation of it in imgui-js. I couldn’t get that to work, and there’s no documentation for it (yet), nor could I find more generalized documentation on how to consume a non-npm TypeScript library.

I also tried to use compile Rust with imgui-rs and winit to WebAssembly. Seems like I was too early on the scene for that idea to work! More on my plans with Rust later in this article.

In the end I had to make my own UI which was a good exercise anyway. I am particularly fond of the very naive window manager I made. Trying to get the UI solutions mentioned above to work definitely took most of my time on this project, and it amounts to nothing in the final project, but I did try some Rust for the first time, and that’s valuable.

Thoughts on the end product

In the end what’s happening is just a bunch of shaders on top of eachother, but with the right shaders and the right parameters you can create some funky effects. Then again, the goal of the project was not create some end all vj tool, but something that evokes the battle backgrounds of EarthBound and makes me better at web development.

I am not much of an actual graphics programmer anyway, my forte is more along the lines of software architecture and user experience. While the current code is a mess, it provides a relatively stable base to build upon. The user experience is also not too bad considering the amount of time I spent on it, if I say so myself.

I really enjoyed building this system that allows me to twist knobs, push buttons, import outside assets, drive itself, etcetera! I also ran it for a bit at a NYE party and people enjoyed it.

Future plans

I am not entirely sure how far I want to take this current project. I met my goal and got everything I wanted out of it. Still, it’d be fun to tinker some more on it, especially if people enjoy it. Here are some things I could do in the near future:

• Refactoring (of course) improving the UI in the process
• Add presets, allow the user to save and share presets (not sure how to the sharing part, sounds like a fun problem to tackle)
• Wrangle the AI more into submission, presets would be useful for that
• Animating values using sines and such, instead of just a random tween
• MIDI controller and/or gamepad support
• Add the ability to use textures for the blend layer instead of just the noise
• Add more different types of textures, geometric and such
• More effects aka more knobs!

Rust

But what I’d rather do is revisit this project sometime in the future when the WebGPU spec is finalized. Rust is becoming more and more viable for web thanks to its first class citizen WebAssembly build support, and WebGPU work is already underway for the language. This would also give me a nice oppurtunity to actually do all the graphics myself, instead of relying on a library such as Babylon.js. For what it’s worth, Babylon.js is also working on WebGPU support; it’s one of the reasons why I prefer it over three.js.

I am very excited about the prospects of Rust on the web. For one, it’s a powerful but also safe language. The ecosystem is great, using other libraries (called crates) is incredibly easy, and the Rust compiler is a good team player that’s very clear in it’s error messages. Another benefit of going this route with a project like this is that it will allow me to target the web, but also target native. This would yield better performance and also maybe allow me to do some things that the web does not permit (supporting NDI, for example).

1. It looks like that Unity is doing something about this with their Project Tiny. Still seems a way off though.
2. I will never write vanilla JavaScript ever again!!