Creating Magic Matta for the 2019.1 Gray Area Showcase

Kieran Lee Farr
A guest pushes a bus controller as part of the interactive Magic Matta project. Photo Credit: Mariah Tiffany

I joined as an artist for the San Francisco Gray Area Foundation’s Incubator Program from January to June this year 2019. I had a great time, met some great artists, shipped a fun v1 of a project, and learned a lot about what it means to be an artist vs. incubate a company. This blog post is a quick overview of the experience and the final showcase.

Why did I join this program?

I have been experimenting with developing experimental VR gaming and non-gaming experiences using the open-source A-Frame WebVR framework for a few years. I was looking for a “forcing function” to combine the inspiration of these previous projects to actually ship something and get feedback from peers and new audiences that weren’t familiar with VR.

The Gray Area Incubator succeeded on all counts — it was a great forcing function to help me ship a project and get quick feedback from peers and the general public.

What is the program?

Gray Area Foundation runs out of the historic Grand Theater in SF’s Mission District

The Gray Area Incubator is a 6-month program run by the Gray Area Foundation for the Arts for beginner and experienced artists alike to develop and showcase new pieces. Gray Area runs out of the historic Grand Theater on Mission St between 22nd and 23rd Streets in San Francisco. It is a unique space with a large event venue downstairs in the converted theater and a multi-use space and classroom in the theater’s former projection room upstairs.

What goes on?

During the incubator I told my friends and colleagues that I was working on an “art project” — keeping things intentionally vague as I wasn’t sure if I’d be able to actually ship anything or that the program would work as intended. But here, finally, I reveal the secrets of what this “art project” entailed… :)

Student perspective of a Gray Area Workshop in progress: a TouchDesigner course taught by Stephen Standridge.

There were about 12 participants, vetted as experienced artists or technically capable of shipping a project at the end of the program. (The “incubator” was not for beginners, there is a separate “Immersive” Program for those looking to learn interactive art technology skills.) Each incubator member had a dedicated co-working desk in the upstairs incubator work-space in exchange for a monthly fee (less than the cost of most co-working spaces in the city). Most folks worked full-time day jobs in addition to their work at the incubator.

Each month the program hosted a group critique led by the program lead Niki Selken. These critiques provided structured conversation — students were expected to come to each critique with a presentation following the format provided by the program to prompt a productive conversation.

For me, that helped to look past many of the short-term technical challenges and consider what the end-product might actually look like and what elements were still missing to see that vision through to completion.

A screenshot from my “Artist Salon” presentation about Magic Matta mid-way through the program. This screenshot shows the HTC Vive Tracker attachment to the model toy bus.

Midway through the program we hosted an “Artist Salon” where each artist presented a quick overview of the motivations and influences behind their work. You can check out the full 15-minute video here, but I’ll summarize much of it below.

Art vs. Profit

Gray Area instructor Gabriel Dunne teaches a photogrammetry class.

I’ve been programmed over the past 10 years in the ways of venture-backed SaaS startups including their tendencies to steamroll over creativity in favor of short-term revenue, and I hold a lot of personal guilt for having steered (or NOT steered) in such a direction many times.

So I appreciated this venture into a new world of “art” and learned a couple of key things, at least new for me:

  • Anyone can be an “artist.” (Or not, and that’s ok too.) It was odd hearing people refer to me as an artist, but I now feel it’s an open term that anyone can adopt upon their own choosing. I don’t think I’ll voluntarily use that word again for a while to describe myself, but I dig the concept a lot more now, feel less guilt about me using it, and less judgment about others using it as well.
  • What does success mean? I struggled with purpose and outcome for this project. What is the definition of success? If it isn’t revenue, adoption, investment, or even social cred, what is it? (My viewpoint after the program is that it’s a trick question when it comes to art: there is no definition of success, it’s up to you as an artist if you even want to ask that question, let alone answer it.)
  • How do you provide and receive feedback? In a world without a definition of success, how do peers share feedback with each other? I really appreciated learning this “through doing” thanks to our Creative Development Director Niki Selkin who ran monthly critiques using a process inspired by Liz Lerman’s “Critical Response Process.” I definitely plan on using this again in the future — it can be applied to a wide range of group feedback settings. (For future reference, here’s a PDF about the process from the Brooklyn Arts Council and a link to Liz Lerman’s own website on the topic.)

What did I work on?

A user lines up the bus controller to pick up “peeps” from the bus stop. Photo Credit: Mariah Tiffany

My showcase project was called “Magic Matta”, an interactive experience to explore transportation and urban planning by encouraging kid-friendly play using the HTC Vive, projection mapping, A-Frame WebXR, and specially enabled toys. Players get down on their hands and knees to push model buses around the play area as controllers for a multi-user game designed for young children to learn counting and the basics of transit systems. (Why the name? “Matta” means carpet in Swedish, “Magic Carpet” is a registered trademark in the US.)

Why did I do this?

Modern VR can be antisocial and physically inaccessible for many people, especially children. Experiences are constrained to start and end inside of an isolated headset and outsiders at best can look on at an external mirror of the viewpoint of the in-VR participant.

A child explores the relationship between the bus controller and the digital projector.

I believe this traditional approach to VR experiences is fundamentally wrong and in this piece I attempt to demonstrate a new paradigm of a persistent virtual experiences independent of any specific viewing device. Participants should be able to interact with the same virtual world in a variety of modes that suit their needs and abilities. Shared physical objects and architecture act as a gateway to ground the connection between the virtual, real, and all the participants.

Magic Matta is a proof of concept that attempts to prove the technology for such a paradigm exists and is accessible by modern developers today to create truly immersive multi-user experiences with standard off-the-shelf hardware and open-source web software tooling.

The VR headset allows an adult user to see the same scene updated in real-time with additional detail and enhanced data overlays.

The “Peeps on the Bus” game concept was inspired by my daughter who plays a similar game on her own to pick up and drop off little wooden people in a school bus. The bulk of technical work for this project was not the game itself, but the infrastructure and tooling to allow such a game to exist. This work is intended to serve as a foundation for other developers to create their own projects that make use of the Magic Matta platform. In other words, Peeps on the Bus, is just the first “app” for Magic Matta and I hope to make (and inspire others to make) many more.

How does it work?

At its core, Magic Matta uses the SteamVR Lighthouse tracking system and HTC Vive Trackers to track the location and pose of up to 16 toys or other input devices. A PC connected to the HTC Vive runs a server to broadcast the location of these devices as well as game state to one or more clients that can display the scene from different viewpoints. One of the display clients is a laptop connected to a projector mounted above and pointing straight down on a white reflective carpet.

A simplified architecture diagram of Magic Matta

The setup enables the carpet to “magically” respond to the movement of the toys shown through environmental lighting effects and interaction with the game using visual guides superimposed on and around the toys.

A once static toy now has interactive lights, sound* and reactivity to a game scene without requiring the child to wear any hardware. Other players can join using more advanced VR or AR headsets, or just pull out their phone and use their browser to experience the same game scene. (*Ok I haven’t implemented sound yet, but it’s coming soon!)

How did people react to it?

A user is surprised to see the bus and scene elements rendered in VR 1:1 with reality, but not her hands.

I appreciated Gray Area’s ability to bring a new audience to my work who provided amazing feedback — sometimes directly and sometimes just by observing how they interacted with the scene.

I heard a lot of “wow” moments when people picked up the bus and saw the reaction to the scene on the floor, and sometimes then again when they looked at the same scene through the headset with the corresponding movement from the bus and virtual objects.

Kids took to the scene more quickly than adults, but credit is due to some adults who got it right away and knelt or sat on the ground and started pushing the bus along the visual guides from the game as intended.

My favorite user testing was with my daughter Avery who inspired the project in the first place. She was able to figure out the game with minimal prompting and her points of confusion (as well as others) are great guides for improving the experience in the future.

My daughter tries out “Magic Matta” at the 2019.1 Gray Area Showcase

Showcase Challenges

The biggest challenge on day 1 of the show was visible lag from using the venue-provided WiFi for broadcasting the server object locations to client viewer computers including the projector. When the highest congestion time of the night came, the venue provided WiFi became overly congested and the project was not useable for up to an hour. We set up a separate router for the next day and that worked much better, even during peak attendance.

VIP-style stanchions restricted the play area to just 1 or 2 people at a time to prevent the tracking sensors from being blocked and interrupting the game. This is a limitation of Steam Lighthouse v1 and the Vive Tracker v1 used in this exhibit.

The exhibit is complicated to set up. It requires 3 simultaneously running computers: 1 PC running the Vive VR headset and accompanying base stations, 2 laptops, a projector connected to one of the laptops mounted on the ceiling pointing down at the floor at a nice right angle to prevent distortion, proper calibration of the playing field and projector, isolated WiFi router that also has a connection to the Internet, custom mounting of HTC Vive trackers to one or more toy controllers, and of course a carpet. (There is also a fair amount of software setup. A more complete explanation of how to set this up yourself can be found in the Github repo.)

When something goes wrong with one component of the setup, it cascades to make the entire experience unusable. Having a quick logo screen to tab over to during downtime helped make the piece look more professional even when things weren’t working as expected.

There were many other complicated technical issues developing the project itself which is beyond the scope of this article, but I look forward to writing more about the software side in a future post and as part of the repo docs.

Down to the wire! Here I am deep in the shadows trying to make a last-minute change during the show.

Was it worth it?

Yes, definitely! Joining the Gray Area Incubator absolutely made it possible for me to complete an immersive art piece and get it in front of an audience.

It was hard. I only had only 1 or 2 nights available each week during the 6-month incubator period as I had a full time job, a wife and young daughter. There were many nights that I didn’t want to leave the house to go to the incubator workspace after a full day of work and family time. If I made the effort to get there, I forced myself to at least commit one line of code to the repo that night.

Much of my productivity was in the last month of the program, especially the very last week leading up to the showcase (highlighted in red below):

I planned ahead wisely to take a few days of work during the final week and it turned out to be crucial to getting everything shipped in time.

It took tons of time. Well over 100 hours over the 6 months, and honestly from time that I didn’t think I had available. My wife did get annoyed at my lack of availability in the last week or so.

Even so, the final project fell short of my expectations. I had expected to be further along in core game design elements including a multiplayer component and additional polish such as sound effects and more effective UI guidance.

What is next for Magic Matta?

The next milestone is to get the core Magic Matta repo in a reasonable state for a stranger with web development and VR experience to be able to deploy this project on their own. It’s technically there, but needs updated and expanded documentation as well as visual and video guides for setup and calibration.

More tracked toys?

Then I’d like to add polish to the “Peeps on the Bus” minigame to have a more engaging multiplayer component and then seek out feedback from more users my daughter’s age as well as educators and game designers, and perhaps reaching out to complementary or overlapping projects like Dynamicland (@dynamicland1) or Tilt Five (@tiltfive).

Please contact me if you have any ideas or interest in the project! I’m @kfarr on Twitter.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade