MRTK x ARFoundation
Doping AR with MRTK
Bringing MRTK Components to the Realm of AR
Holla peeps,
I am a huge supporter of Immersive Tech, and I strongly believe that it is the future of Human Computer Interaction. Although, immersive Tech consists of Virtual Reality, Augmented Reality and Mixed Reality, but most of us have been working in the space of Augmented Reality due to the readily available mobile phones.
Recently I got a chance to work on Mixed Reality, i.e. develop for Hololens 2 using MRTK, and man, I was blown away by the possibilities it has to offer. I instantly fell in love with Mixed Reality and it actually gave me sense of Iron Man stuff(all those who have tried Hololens 2 can easily relate with me).
I got to know about how we can use MRTK for AR development as well. I was really excited to give it a try and so I did and well it worked like magic.
Through this blog, I will be sharing how we can we bring the power of MRTK in the realm of AR.
But first of all let us get to MRTK and ARFoundation, then we can understand how and why the combination of both can be useful.
What is ARFoundation?
AR Foundation allows you to work with augmented reality platforms in a multi-platform way within Unity, .i.e while making an AR app for mobile device, we just need to develop using ARFoundation and then ARFoundation will automatically use ARCore/ARKit depending on the device the application is running. So with a single code base we can build our AR app for both Android as well as iOS. Isn’t that great???
What is MRTK?
MRTK stands for Mixed Reality ToolKit. It is a Microsoft-driven project that provides set of components and features, used to accelerate cross-platform MR app development in Unity.
So now that we have a basic idea about MRTK and ARFoundation, let us take a look what we will be doing?.
We will using MRTK along with ARFoundation
What good will that do? By doing this, we will have access, to the powerful features and components offered by MRTK, and we will be able to use them while making cross-platform AR apps.
Still not convinced. Check out the below listed specific use cases:
- MRTK’s Object Manipulator makes an object movable, scalable, and rotatable (way better than LeanTouch(the only choice you have in AR))
- MRTK’s UI Building Blocks provide great pre-built components for Spatial UI (way better than Unity UI Elements)
- MRTK’s Object collection allows to lay out an array of objects in predefined three-dimensional shapes
- MRTK’s ToolTip support lets you easily add tooltips of 3D objects
- and many more…..
To do the above mentioned things in AR you would have to use different plugins (if available) or write your own. But with the power of MRTK it is easy pizzy, and the best part is that you have many more such useful features in MRTK and the list is progressively increasing.
So now that I have convinced the AR developer inside you that MRTK is indeed useful,let us cut the chit-chat and get started.
Steps to complete
#1) Project Setup
- Creating a new Unity Project in 2019.3.x
- Adding ARFoundation Support
- Adding MRTK Support
#2) Collaborating MRTK with ARFoundation
- Creating a very basic scene
- Running MRTK Examples using ARFoundation
Project Setup
In this part, we will be creating a new Unity project and adding all the dependencies.
Creating a new Unity Project
Simply create a new project in Unity (use Unity 2019.3.x) with 3D Template and name it whatever you want to, I will be naming it “MRTK_ARF”.
Now that our newly created project has been initialized, let me change the layout of my editor to “Tall”, because that is how I like it.
A quick suggestion from my side to all the Unity beginners to change your Unity Layout to “Tall” if you are beginner in Unity, so that you can follow my steps easily.
Let us also create a good folder structure so that the things don’t get messy afterwards. We will make a folder named “MyProject” which will contain all the scenes,scripts,prefabs,etc created by us.
Adding ARFoundation Support
In this step we will be adding ARFoundation via Package Manager.
Simply go to Window->Package Manager. Then in the Package Manager Window select “All Packages” from the top left corner and select “Show Preview Package” from the top right corner. Once all the packages show up, search for the following packages and install the specific packages(Checkout the gif given below.):
- ARFoundation (v 2.1.4)
- AR Subsystems (v 2.1.1) (This will be automatically installed while installing ARFoundation)
- ARCore XR Plugin (v 2.1.2 )
- ARKit XR Plugin(v 2.1.2)
Adding MRKT Support
Next step is to add MRTK Support so that we can use the powerful features and components. Now , adding MRTK Support is not easy as installing a package from Package Manager, but it is not that difficult as well. We just have to download Unity Package of MRTK from MRTK GitHub releases. Go to the MRTK release link given below and download the following:
- Microsoft.MixedReality.Toolkit.Unity.Foundation.2.3.0.unitypackage
- Microsoft.MixedReality.Toolkit.Unity.Examples.2.3.0.unitypackage
Once you have downloaded both the above mentioned packages, now import both the packages, one by one, in Unity. To do so go to Assets->ImportPackage->CustomPackage and choose the packages one at a time. (Checkout the gif given below for your help.)
Once you add both the packages, you might see some warnings in the console, feel free to ignore them.
Now first part of this blog is done. Hurray….? Not really!!!
Collaborating MRTK with ARFoundation
We have setup everything in place and now its time to have fun.
Creating a very basic scene
So, for this step, create a new Scene and save it as “BaseScene” in the Scenes Folder that we made while making the Folder Structure.(.i.e inside Assets/MyProject/Scenes).
Now, we will be adding the MRTK Support to our BaseScene and Configure it to use MRTK. This is as simple as going to Mixed Reality Toolkit->Add to Scene and Configure in the Menu Bar.
Once you configure your scene with MRTK, your scene will we configured with Default MRTK Profile Settings. You can see the Profile by clicking on the MixedReality Toolkit in Hierarchy and checking its properties in the Inspector Window.
So before moving forward, let us understand what are Profiles in MRTK?
Profiles are one of the main ways in which we can configure MRTK as per our use case. The top level MRTK Configuration Profile contains Sub-Profile data for each one of the primary core systems, each of which are designed to configure the behavior of their corresponding sub-systems like Camera, Input,etc. MRTK comes with a Default Profile, with pre-set default option. This gives user a head start while creating an experience using MRTK. The user can easily edit the Profiles as per his/her own use case. Although, the user cannot directly edit the Default Profile, the user has to create a copy of the Default Profile and then do the required changes to the duplicate profile. This is done so as to preserve the Default Profile Settings for cases when user wants to come back to Default Profile.
For our use case also we will be creating a copy of the top level MRTK Configuration Profile and also of the Camera Profile(as we want MRTK to use the mobile camera using ARFoundation).
Want to know more about MRTK Profiles, then checkout the link given below.
It is time to create our own custom MRTK Profiles. This is pretty easy, as everything else in MRTK.
Step #1: Creating a Custom top level MRTK Configuration Profile
Select the MixedReality Toolkit Object in Hierarchy and in the Inspector window click on “Copy and Customize” and save it with name “MRTK_ARF ToolkitConfigurationProfile”. You can find the newly created profile in Assets/MixedRealityToolkit.Generated/Custom Profile Folder.
Step #2: Creating a Custom Camera Profile
Now that we have created a top level Custom MRTK Configuration Profile, now we can edit the sub-profiles. Navigate to the Assets/MixedRealityToolkit.Generated/Custom Profile Folder and select the newly created top level Custom MRTK Configuration Profile (named MRTK_ARF ToolkitConfigurationProfile). Now in the Inspector,select the Camera sub-profile and click on “Clone”. Save the Custom Camera Profile with a name “MRTK_ARF MixedRealityCameraProfile”.
Step #3: Setting up the Custom Camera Profile for using ARFoundation
Next step is to edit the newly created Camera Profile. We will be removing the Windows Mixed Reality Camera Setting and will be adding Unity AR Camera Settings. Checkout the gif given below on how to achieve the above mentioned.
Step #4: Build the app
Before building the app check the following things:
- MixedReality Toolkit Object in your Hierarchy is set to use the Custom MRTK Top Level Configuration Profile (created by you in Step #1 by the name of MRTK_ARF ToolkitConfigurationProfile).
- Also, that the Custom MRTK Top Level Configuration Profile is using the custom created Camera sub-profile named (created by you in Step #2 by the name of MRTK_ARF MixedRealityCameraProfile).
- Switch platform to Android/iOS before building the app (Here I will be trying out on Android).
While you will do so MRTK will prompt you with platform specific setting, make sure to Apply those settings.
- Now if you try to build you will get error related to Vulkan Graphics API as ARCore doesn’t support it. So we will resolve this by removing Vulkan Support by going to File->Build Settings->Player Settings->Other Settings->Rendering and will the remove Vulkan Support. (Check out the gif below.)
After doing this you can go ahead and build the app.
Once you have successfully built the apk and you run the application you will nothing in the scene but a console type window following the camera (as shown below).
In case, you just see the “Black Screen” , then check out the Bug #1 given at the end of this post.
Now, probably you are wondering that we did all this to watch a console window follow us. Ummm… Absolutely not. By making this base apk we made sure that everything is working Perfecto!!!. Now let us move on to next step, which I am sure will make you happier (God Promise !!!…xD).
Running MRTK Examples using ARFoundation
As everything is working great but you still don’t seem to be happy, we will be running MRTK example and showcase you what all could be achieved with the support of MRTK.
For this we will building the HandInteraction Example Scene.
Step #1: Go to Assets/MRTK.Examples/Demos/HandTracking/Scenes and open up the “HandInteractionExample” Scene. If you are prompted to install TMP (TextMesh Pro) then install TMP Essentials.
Step #2: Make sure that MixedReality Toolkit Object in the Hand Interaction Example Scene is using your Custom Top Level MRTK Profile (named MRTK_ARF ToolkitConfigurationProfile).
Step #3: Go to File->Build Settings and remove any other scenes and add only the HandInteractionExample Scene.
Step #4: Build and Run the app and experience the power of MRTK. (It will take some time to build, so be patient.)
Output of your apk should look something like the video given below. Feel free interact with objects.
I am sure that now you would be happy to see the capabilities MRTK has to offer. I suggest you to try out different MRTK example as in doing so you will get to know about different features that MRTK has to provide.
Although I could have easily finished the blog here but I have something more to offer to peeps who want to dig in more. This is a bonus section and doing this you will get to know how easy it is to create experiences using MRTK.
The Bonus One
In this section we will be doping (is that even allowed here?) ARFoundation experience with MRTK Components. We will be making a simple scene in this we will be addressing the “Move-Scale-Rotate”.
In most of the AR experiences we want to give the user the ability to interact with 3D objects via Moving, Scaling and Rotating. Most AR creators will agree with me when I say that most of us are using LeanTouch for adding this Move-Scale-Rotate functionality. In this section we will be using ObjectManipulation Component offered by MRTK. While doing so you will notice how effortless it is to add functionality and yet how amazing results it gives.
Step #1: Add ARFoundation Support
- For this you will be adding AR Session Origin and AR Session by Right Clicking in Hierarchy->XR-> AR Session Origin and AR Session.
- Now delete the AR Camera situated under AR Session Origin.
- Now drag and drop the MixedRealityPlayspace/MainCamera on to the missing prefab of AR Session Origin Script on the AR Session Origin Object.
- Next we will be adding Plane Detection and Ray Casting. We can do this by adding ARPlaneManager and ARRaycastManager Scripts on AR Session Origin.
To see (visualize) the detected planes we need a prefab. We can get this prefab by Right Clicking in Hierarchy->XR-> AR Default Plane. Now drag this AR Default Plane prefab and save it in the MyProject/Prefabs folder. Now set this prefab in the Plane Prefab property of ARPlaneManager Script.
- Create a new script named “PlaceOnPlane” in the MyProject/Scripts Folder. Then copy the script given below and paste it in your script in Unity. This is a basic script which detects plane and instantiates user defined prefabs. (Check gif given after next point)
- Add PlaceOnPlane Script on AR Session Origin. (Check gif below.)
Step #2: Creating MRTK doped prefabs to be instantiated in AR
- For this part we will be needing a few 3D models. You can easily get those from sites like Poly or SketchFab. Or you can easily use a 3D primitive cube available in Unity (If you are using Cube make sure you scale down your cube to something like “0.2,0.2,0.2” ).
- Add ObjectManipulator and BoundingBox to the 3D Object.
- Set the BoundingBox properties as shown in below image.
- Now save the Cube as a prefab in MyProject/Prefabs Folder.
- Create a few more 3D Objects following the points given in Step #2.
- Set the prefabs to be instantiated on PlaceOnPlane Script attached to the AR Session Origin.
- Build the app.
Below video shows use of BoundingBox in ARFoundation.
ENDing Notes
In the end of this blog, I hope you got to learn a thing or two. I hope you will use the powers ,offered by MRTK, responsibly. I am attaching the GitHub repo link of the complete project. Feel free to download it make changes according to your needs.
Bugs and Solutions
Bug #1: The apk builds properly but results in a “Black Screen”.
Solution #1: In File Menu Bar, go to Mixed Reality Toolkit->Utilities -> UnityAR-> Update Scripting Defines.
Bug #2: MRTK Materials are not updating to Light Weight Render Pipeline.
Solution #2: In File Menu Bar, go to Mixed Reality Toolkit->Utilities -> Update MRTK Standard Shader to Lightweight Render Pipeline.
Feel free to connect with me on LinkedIn.
Kudos to everybody
-Jking
(Jatin Pawar)
XR Enthusiast
