(more on “mixed reality”)

I’ve recently been asked by several developers about using the HTC Vive’s front facing camera. Questions such as “How do I implement SteamVR’s Tron view in my own app?” or “How do I use WebCamTexture or Texture2D with Unity and [Vuforia or OpenCV or other solution] ?”

Let’s get a basic working implementation of a front facing camera filter like this black and white comic filter example found in OpenCV for Unity.

Image for post
Image for post
front facing filter fun

You can download the executable here to see the filter in action.

For camera access, there are two approaches with Unity: Texture2D or WebCamTexture .

Texture2D with TrackedCamera

In a previous article I’ve covered the basic use of the front facing camera (also referenced as TrackedCamera) by using the OpenVR API which we mapped to a Texture2D texture in Unity for a heads up display example (and since then the SteamVR Unity plugin also provides a similar example). …

Another use case for mixing up reality

In part one I reviewed how “Mixed Reality” is being used to describe several use cases, one being the use of a green screen for creating a view of the person experiencing VR from within the virtual world.

That use case is supported with the SteamVR plugin for Unity without requiring code but I also provided some workaround code if you’re unable to obtain a third controller (see part one).

Another use case for MR or technically AV (Augmented Virtuality) is the use of the front facing camera on the Vive:

Image for post
Image for post
controllers in front of a live camera view of them

As first introduced at CES earlier this year, when you enable the camera you can see a threshold based contour of what the camera is seeing. The initial demo included placing a real physical chair in your play space so that you would be able to sit down without taking off the headset. …

What is Mixed Reality (MR) ?

There’s been much confusion about and misuse of the term “Mixed Reality” lately. Even the Wikipedia entry for it notes that it needs a complete rewrite. I’d like to contribute some clarity and provide a how-to for some of the use cases.

I used to post the following graphic in my presentations on VR last year. It helped contextualize the discussion about VR/AR with the historically correct taxonomy:

Image for post
Image for post
This is the “Reality-Virtuality Continuum” graphic by Paul Milgram, et. al. created in 1994 (wiki entry)

A specific misuse of “Mixed Reality” is the use of a green screen for VR streaming or promos (see the image below). Technically this could be described better as augmented virtuality (AV). In this use case an external camera and a green screen are used to mix the “Real Reality” view of the player into the virtual world that player is inhabiting. That does sound like a use for “Mixed Reality” except that as shown above the proper definition of MR is a super set of VR and AR and things in between. …


Dario Laverde

VR/mobile developer, community leader and sr developer evangelist at htc

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store