Exploring AR through Microsoft Hololens

What if I told you that your favorite childhood super hero was engaged in an epic battle right behind you and your room was an active battleground?

Would you believe me?

Just a few decades ago, asking this question to the typical consumer might earn me a response such as “Are you okay?”. Today, however, through the latest advancements in computer vision and wearable technology, we are able to glimpse at an eventual future where digital creations and physical realities can coexist. This future has a name: Augmented Reality.

What is Augmented Reality?

Just like the name suggests, Augmented Reality is our physical reality enhanced (or “augmented”) by digital creations of various forms (sound, video, graphics, animations). The term was coined by Thomas P. Caudell (former Boeing engineer) back in the 1990s when AR was still relatively unknown outside of the research community. Through advancements in computer vision, object recognition, and wearable technology, Augmented Reality has evolved from a simple virtual cube in the 1970s to interactive and digitally manipulable applications that are overlaid on the real world.

Augmented Reality vs. Virtual Reality

One of the most common misconceptions consumers have is the difference between Augmented Reality and Virtual Reality. Simply put, Virtual Reality is a fully immersive experience inside a digitally generated reality whereas Augmented Reality consists of digitally generated content that is superimposed onto the physical reality. The uses for both AR and VR are ample but it is predicted that the AR market will be much larger in size due to its applications in everyday life. While Virtual Reality headsets (HTC Vive, Oculus) have already hit the consumer market, AR headsets have yet to become majorly available to everyday consumers. This makes it a very lucrative market and skill set to acquire for interested developers.

Source: Mirchu

Introducing Microsoft Hololens — AR Headset

Over the past decade, companies and researchers have been hard at work to bring Augmented Reality into the hands of consumers. One of the key companies that looked into this technology was Microsoft. This year, Microsoft released an AR wearable called the Hololens. It is the first AR wearable development kit supported by a major technology company.

Source: Microsoft Hololens (Youtube)

Developers from around the world (myself included) have collectively dreamed up thousands of applications for AR using this development kit. The community for AR has been developing and growing exponentially through the use of the Hololens.

Why Microsoft Hololens over competitors?

While there are many other AR headsets out in the market right now (Meta 2, CastAR, etc.), Microsoft Hololens is the only AR headset that is led in the development efforts by a major technology company. This means Microsoft will use its wide spanning resources, reach, and talent to make sure Hololens can receive a good share of the AR market in the future. Due to this, it is an extremely lucrative platform for first time developers to embark on their AR programming adventures. Additionally, with the funding that Microsoft provides the Hololens and the dominant marketing strategy that Microsoft is following, this platform has gained the largest and most loyal developer community in the early stages of the consumer AR market. This can directly translate to a larger application base as well as community support and modifications to existing Hololens features. This really contrasts Hololens competitors such as Meta 2 and CastAR, who has less of a supporting community and is lesser known due to their start-up status.

Microsoft Hololens — A Beginner’s Tutorial

Do you want to make a super hero battle in your room a reality? I will now provide a starter guide to programming for the Hololens. Let’s go!

  1. Getting a Hololens Dev Kit

While you do not need a Microsoft Hololens Dev Kit in order to start developing for it, having a physical Hololens is certainly fun and exciting! You can purchase a development kit on the official Microsoft Store (link).

Note: If you decide not to use the physical device, just use the Hololens Emulator. We will be using the Hololens Emulator in the tutorial.

2. Setting up your development environment

First, make sure you have either Windows Pro, Education, or Enterprise edition. This is needed for a virtualization feature (Hyper-V) that is only available in those editions. Note: Without this feature (Hyper-V), you will not be able to run Hololens Emulator!

Second, follow the links provided below and make sure you have correctly installed all of them.

Note: These are the essentials to get started. A detailed version of tools for the Hololens can be found here.

Third, ask for help on the forums if unexpected errors occurs during the installation process. Different systems might run into unforeseen issues that might have already been investigated in the forums. A link here.

3. Creating the “Hello World” of Hololens

This section is about creating your very first Hololens application! While it is not comprehensive, it will encompass the core steps that will get applications running on the Hololens Emulator or the Hololens itself. I assume the reader of this tutorial to have no Unity knowledge or Visual Studio knowledge.

  • Let’s begin! First open Unity for Hololens (Installed in the previous section). You should see the following screen.
Open Unity (Hololens) and then press “NEW” to set up a new Hololens Application
  • Name your project “HelloWorld” and make sure “3D” is selected. Then press “Create Project”. (Shown below)
Click “Create Project”
  • On the upper left hand corner of the now visible Unity window, make sure that “Layout” is set to “Default”. If not, change it now.
Change “Layout” to “Default”
  • Now we will set up the Augmented Reality camera view for Unity. Let’s expand the “Untitled” project and reveal the “Main Camera” and “Directional Light”.
Expand “Untitled” to reveal “Main Camera” and “Directional Light”
  • After double-clicking the “Main Camera”, you should see the following on the right-hand side of the Unity window.
Double-click “Main Camera” to reveal “Inspector” on the right-side of the screen.
  • We will now change a few of the values in this “Inspector” window. Change the “Position” variables to X: 0, Y: 0, Z: 0 so the camera (head of the user) originates from the origin frame of the application). Change the “Clear Flags” field to “Solid Color”. Click on the “Background” and change it to a solid Black color with RGBA values being 0,0,0,0. Finally, change the “Clipping Planes” field to 0.85 based on Microsoft recommended settings. Make sure your “Inspector” window looks like the one show below.
Change “Position”, “Clear Flags”, “Clipping Planes”, and “Background”
  • Next, we will create a 3D text to show up on your Hololens! Let’s go ahead and click “GameObject” in the tool bar. Then, we will click on “3D Object” and then “3D Text”.
Click on “3D Text”
  • Next, we change some attributes for the 3D Text. In the “Inspector” on the right, you should see this. The changes that were made are as follows: Change “Position” to X: 0, Y: 0.6, Z: 5. Change “Scale” to X: 0.5, Y: 0.5, Z: 0.5. In the Text Mesh change to what ever text you want (“Microsoft Hololens Hello World” in this example. Change “Anchor” to Middle center and change “Alignment” to Center. Change “Color” to which ever color you like (red in this example).
Change 3D Text attributes
  • The window should look like this after all of the 3D Text attributes have been selected. Note: The view have been changed to “Game”. It is boxed in red below.
Final view before saving
  • Now we are ready to save this Unity file! Click “File” and then click “Save scene as”. On the prompted save screen, save the file as “Hello” in the Assets folder of the app.
Save Scene as…
  • Next up, we will build the application to load into Microsoft Visual Studio 2015. Click on “File” and then on “Build Settings”.
Build Settings
  • There is a few items we want to do on the Build Settings screen. First, click “Add Open Scenes” to add the current scene “Hello” to the Scenes In Build. Next, make sure “Windows Store” is selected under “Platform”. Then, ensure that “SDK” is set to “Universal 10” and “UWP Build Type” is set to “D3D”. Also, check the “Unity C# Projects” box. After all of that, click “Player Settings…” to make a final change to the file.
Build Settings
  • Under “Player Settings Inspector”, switch to the Windows Store View as shown below. Then, make sure to check the “Virtual Reality Supported” button. It should say Windows Holographic under the SDKs.
Player Settings Inspector
  • Now, you can build the project! Just go back to the Build Settings and click Build.
  • When asked to save the project in a folder, please create a new folder called “App”. Then select the folder by pressing “Select Folder”. As shown below:
Make a new folder called “App”
  • Once the project has compiled, you should see the Visual Studio 2015 Solution File inside your HelloWorld/App folder. Double-Click it to open up the solution inside Visual Studio 2015.

Note: Common mistake here is to use the solution from the main HelloWorld folder. Don’t use that. Use the one above in the HelloWorld/App folder.

  • Once Visual Studio 2015 is open, we are going to change some settings so we can run our application. At the top of the screen, change the first option to “Release”, the second option to “x86”, and the third option to “HoloLens Emulator”. After this, just click the Green PlayButton to run the application!
  • Here is where some might run into a few troubles. If prompted with this screen, just click “Retry” and then “Yes” to the additional prompt. This is to give Hyper-V administrative rights.
Click “Retry”
  • The next thing you should see if the Hololens Emulator start up. With OS Starting… looking like this:
Hololens Emulator
  • After waiting for a 2–10 min (based on your computer speed), your application should be loaded! Initially it should look like this:
  • Controls: You can move the camera using your mouse. Also, you can zoom in (Shift + “W”) or zoom out (Shift + “S”).
  • CONGRATS! You have finished your HelloWorld Application for Hololens Emulator!

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

More Features of Microsoft Hololens

Now that you have built a HelloWorld project for the Microsoft Hololens, you know the basics of Hololens app development. However, there are a lot of other features that the Hololens has. We unfortunately could not cover all of them. Here is a quick list of some of the key features that could be used:

  • Spatial Location Memory — (Used to remember the location of scanned objects, walls, surfaces as well as pinned applications. Very useful for games and many other cool applications.)
  • Gaze Tracking — (Used to track eye movement and where the user is looking. This helps the digital world adjust to your view.)
  • Gesture Input — (Used to recognize common gestures (tap, bloom) as input devices in many applications. These can be programmed into your application to provide more user interaction.)
  • Voice Support — (Voice can be used to give commands and provide input. For example, you can program your application to accept certain voice inputs for certain actions. “Hey Cortana” is the classic example.)

To read more on these features as well as how to properly use the Microsoft Hololens Tools, visit the forum!

Microsoft Hololens Example Project — UMN Coin

As an example, I have created a project using Vuforia (an AR Platform). Here I use Spatial Location Memory to retrieve the location (as positional data) of my UMN coin with respect to the recognized image. When I switch the image, my coin disappears and another coin replaces it. It also uses Gaze Tracking to make the coin move more fluidly to the user. A quick gif of it can be seen below:

UMN Coin

Due to this example project being a longer process than the HelloWorld project, I did not attach a tutorial on how to create it. However, I might update this article in the future with a link to the GitHub repo that contains the source codes for UMN Coin.