Immersive Experiences & Messy AR

Giving old tech new life

This article documents part of a 2 week long course on Immersive Experiences facilitated by James Tichenor and Joshua Walton at the IDP 2019 at the Copenhagen Institute of Interaction Design (CIID)

From our Friday exhibit

We take a lot of diverse courses here at CIID, but few are as highly anticipated as Immersive Experiences. For most of us, it’s our first time playing in the sandbox of Unity, Vuforia and Spacebrew — the possibilities were endless.

Our group (Mitsu Shah, Pooja Pawwar and Gaana Srinivas) chose to investigate microinteractions in the context of virtual/augmented/mixed reality, and had one week to create a project around the subject.

What makes a good (micro)interaction?

To start, we did some desk research to find products, installations and experiences that were driven by a single interaction, and explored what made them delightful and worth revisiting.

Here’s what we found. A good microinteraction:

  1. Can be pleasing, compelling and/or delightful
  2. Directs the user’s attention
  3. Effectively communicates affordance to the user
  4. Provides (instant) feedback about a completed action
  5. Completes the do-feel-know (interaction) loop

Obsolete, but not forgotten

When’s the last time you used something with a bunch of buttons? No, not UI buttons, we’re talking actual physical buttons.

There’s so much to love about a cassette player. The crisp snap of closing the door, the shhh of a tape rewinding, the mechanical clunk of a real button that’s so satisfying to press and release, again and again…

Some people claim they can hear this GIF

Before the digital era, tech was plasticky and things sometimes needed a good whack to get working again. Tech used to be messy, and we used to love its quirks.

Augmented reality is rarely messy. 3D objects rendered in conventional reality are often unnaturally pristine surfaces, at odds with the world around them.

Obsolescence + AR = Profit??

We had the beginnings of a good microinteraction in the lovely tactile cassette player, which we dug out of a box of junk in the lab. We decided to give it new life — these controls would pilot a totally virtual video player.

Like this, but real! As real as AR is to you, anyway.

Hardware — Makey-Makey, Sony Cassette-Corder & Copper Tape

The three of us were far from any sort of engineer. But by studying the mechanics of the cassette corder, we were able to get it to work in our favour in several steps:

  1. For proof of concept, we wired the buttons externally with copper tape to the Makey-Makey which was connected to the Macbook. This way we could assign the cassette corder buttons to perform actions on the Macbook.
Imagine a Makey-Makey with a bunch of jumper cables connected to it off to the left

2. Upon taking a closer look, the mechanisms of the cassette-corder that played and stopped the tape were accessible through the cassette door. We wired this mechanism with bits of foam and copper tape to make contact switches we could reconnect to the Makey-Makey. This meant users would be able to operate these buttons (devoid of their copper embellishments) as originally intended.

3. For the final product, all the wiring was packed neatly into the body of the cassette corder, with a handy bit to control the volume also concealed into the side. It’s like we didn’t even mess with it.

…When the cassette door is closed, but how else would we show you all the hot glue and copper tape that runs our projects? You’ll see later, it’s amazing.

Software — Vuforia, Unity, C# and maybe some Python?

Pictured here: someone who got Unity to work and was too excited to wait to show her teammates

In addition to learning Unity and Spacebrew the previous week, we also used Vuforia to bring our experiences from the virtual world into ours. We used YUXI — the Mixed Reality Hardware Toolkit by Sensory.cc to kickstart our journey.

As with any project, we took baby steps and kept building.

  1. We used YUXI sample code to map a sphere’s Z value to move when the user touched the wire connected to the Makey-Makey, which was mapped to the Macbook’s keyboard.
  2. A few Youtube tutorials and much confusion around digital materials and shaders later, the video was then playable within Unity. It’s a Jonathan Pillows video by Adult Swim. It was chosen to be as odds with the cassette-corder as possible.

3. Finally, we assembled the rest of the visual experience — the player’s frame and the floating objects, and focused on making it responsive to the user’s actions on the physical buttons. An interesting point of discussion was around where the fiducial ought to go.

Our video player in Unity, with the fiducial at the bottom

Fiducial [ fi-doo-shuh l] n. — A fiducial marker or fiducial is an object placed in the field of view of an imaging system which appears in the image produced, for use as a point of reference or a measure.

Part of our challenge was to naturally incorporate the fiducial into our experience. So we ended up sticking the fiducial on the cassette-corder (where else would you put it anyway)

Which meant that our video player would ‘stick’ atop our cassette corder in virtual space, making it truly portable once again

Watch the video!

Thanks for sticking around long enough to read this! Watch our video and you’ll regret having spent the time you did on processing words, like a peasant.

You can find the team here:

Gaana Srinivas — Website, Medium, Dribbble
Mitsu Shah — Website
Pooja Pawwar