Aquatic Life Inside the Looking Glass
a postmortem by Fernando Ramallo & Kevin Watters
We’re Fernando and Kevin — artists and game developers based in Brooklyn and creators of A Very Holographic Aquarium, an interactive aquarium environment made exclusively for the Looking Glass.
Here’s an overview of our experience making this project and working with the display for the first time!
Conception
When Looking Glass Factory first reached out to us as fans of our previous project PANORAMICAL — as well as our Unity tools Doodle Studio 95 and Raymarching Toolkit — we were excited to check out the hardware and talk about a possible collaboration.
After looking at some existing work made for the display, one idea kept coming back to us: a living, dynamic aquarium of creatures and vegetation contained within the Looking Glass, complemented by the device’s glassy finish and reflective surfaces.
As a first step, we decided to start with familiar tools from our previous projects and a design similar to PANORAMICAL to prototype something beautiful quickly while getting acquainted with what worked and didn’t in the Looking Glass.
We then started putting together some art direction references, looked at a bunch of aquariums (aquaria?), and did some quick research on how we’d go at implementing our ideas.
Implementation
The Looking Glass SDK for Unity was straightforward to work with. All it took was for objects to show up in the Looking Glass by dropping a prefab into our scene in Unity, which became immediately visible on the display without much configuration, letting us focus on creating a visually cool environment.
Laying out the scene
We spent quite a bit of time at the start figuring out how to lay out our scene to work best in the display. A few questions immediately came to mind:
- Is it an aquarium or a fish tank?
- Is the world contained within the display or are we seeing a window into a larger space?
- Are we looking forward or down?
- Is the camera moving or static?
- Are we looking at a large valley or a small pile of sand?
A rough layout of the scene with a simple terrain and some box-shaped fish to try out different configurations of camera positions and scale helped us answer these questions quickly.
We soon found that a fixed camera position, rotated at the user’s control, with elements arranged loosely radially from the focus plane outwards worked best for what we were trying to build.
The team at Looking Glass had a few great suggestions — the camera could face down at 30 degrees to match the ground to the surface the display is sitting on. They also suggested a Depth of Field effect to gently blur out what’s outside the focal point, and additionally to keep objects from getting too close to the camera to prevent headaches or eye strain. These tips really proved important in getting our prototype to “pop” inside the Looking Glass.
Developing the Assets
Flora and Fauna
We wanted to let users interact with lots of coral-reef-inspired vegetation and creatures that swim between them, all the whilst making everything feel very physical and responsive.
The main vegetation was made by creating a chain of colliders bound together by Spring Joints, like a wobbly chain. Then, a Tube Renderer draws a tube mesh from joint to joint, resulting in a soft tube that responds to physics.
Because it ended up looking and and feeling really good in the Looking Glass, we ended up reusing the same system for the fish and some flat vegetation, too.
From the start we were mesmerized by the big patches of grass from the reference material. We implemented a few foliage types based on chenjd’s grass and made it stick to the terrain and adjust to the land’s slope in a custom shader.
Art Direction
We didn’t want to go for a totally realistic approach but instead have a space that feels grounded, showcases the display’s depth, but is at the same time whimsical, colorful and playful.
By carefully tweaking and iterating on the colors, we arrived at a palette of pale blues and cyans for the ground contrasted with bright strong reds, greens and oranges for vegetation.
The lighting setup includes a bright coral reef rig that smoothly transitions to a dark deep underwater bio-luminescent forest. We do this by having a global shader value that all materials use to blend between two gradients or textures.
We found that tweaking the colors on the device whenever possible was essential since they look quite different on a 2D screen.
Also, flat colored objects, just like in VR, don’t read well since you can’t tell how far into the scene they are. We had to keep some amount of lighting and shadows to keep their depth readable.
Interacting with a hologram
An initial question was — how does one even interact with an application made for the Looking Glass?
A few existing demos in the Looking Glass Library use the Leap Motion controller, but we wanted to try and tackle more direct type of interaction that would allow for tons of playful possibilities.
We had a MIDI Fighter Twister controller around and thought it’d make for a great and welcoming showcase piece to control the Aquarium with it. Each knob would smoothly modify the scene by changing individual elements such as the creatures, bringing in foliage or changing the light.
Because we knew that not everyone would have a MIDI easily accessible to them, we programmed the Aquarium to also allow players to use a regular keyboard and mouse setup by holding a key and moving the mouse to mimic the behavior of the knobs.
Finally, we added support for the Looking Glass’s face buttons so you can fully experience the Aquarium without any external hardware.
Holographic UI
Because the interface is a fairly new one, we wanted some sort of tutorial to teach users the controls, so we had to figure out what a holographic user interface would look like.
After putting a great deal of work into creating a space that felt physical and leveraged the display’s depth, we worried that simply placing floating text on top of everything wouldn’t look so great, and might puncture the illusion of the three-dimensional visuals.
To display an interface in a way that fit with rest of the scene, we arrived at the idea of displaying text on a smartphone that accidentally fell into the aquarium, forever lost to its owner.
To keep the text readable on the display’s resolution, the font needed to be bold and in a larger size than usual.
Almost-infinite possibilities
With the Looking Glass display being such a nice object to have on a desk by itself, like a 3D picture frame, we wanted the Aquarium to be something you could leave running and where every time you look at it you see a new combination of elements, slowly shifting into one another.
We added a script that — after a few seconds of inactivity — slowly shifts the scene towards a new configuration of elements.
The new configurations aren’t totally random, but chosen following certain rules to keep the combinations from looking all the same: Pick one type of terrain, one type of grass, maybe fish and maybe change the light, and randomize those within a certain range while keeping everything else off.
Performance considerations
Since the Looking Glass display relies on rendering the scene many times from many perspectives for every frame (45!), we quickly realized performance was going to be something to keep a close eye on.
The Aquarium doesn’t need quick interactions and the elements are designed to shift slowly, we could get away with lower frame rates than you would typically see with an action game. Nevertheless, we kept things manageable by keeping a eye on the profiler, simplifying our shaders and making use of GPU optimizations like instancing, as if we were making a mobile game.
Draw calls can really stack up on the Looking Glass, since they are multiplied by the number of views, so for our project we found it useful to use Unity’s Mesh.CombineMeshes function at the beginning of each frame on sets of meshes that share materials, but weren’t being batched automatically, to keep the draw call number low enough.
Conclusion
Working with the display felt like working with the first VR headsets: a whole new way of thinking about crafting and interacting with 3D worlds. It was interesting to notice how a lot of the lessons we learnt from VR applied to the Looking Glass.
Its depth and size made us rethink a lot of the usual design considerations from 2D screens. It was super interesting to tackle these problems and arrive at a piece that explored this territory a bit further.
Finally, a big shoutout to Corey Bertelsen for his audio work which brought everything up 1000%!
Hope this write-up gives you a few guidelines when tackling your own projects in the Looking Glass! Can’t wait to see what folks make with it!
Get A Very Holographic Aquarium for your Looking Glass on the Library. Download that here.
A Very Holographic Aquarium is designed by Fernando Ramallo and Kevin Watters. For more information about the Looking Glass Factory, visit this page.
