Doodle_AR — Bringing your doodles into the realm of AR
Hello peeps,
So one fine day (that was yesterday though), I was hit by an idea of trying to combine the power of OpenCV and the magic of AR. In the process of doing so, I set myself on the quest to find something which was bigger than us i.e. DoodleAR.
I did some to google-woogle to check if somebody has ever worked on this before, and to no surprise I found out that Takashi Yoshinaga has already some great work in this field. So, this is a blog on taking things up from where Takashi left. I found out the tracking could be greatly improved. So I thought of improving the project and also writing a blog explaining the in-depth details so that people can learn from it.
A big shout out to Takashi Yoshinaga.
You can checkout the original work by Takashi Yoshinaga here.
He is doing really great work in his field, I suggest everybody to checkout hiss work here.
In this post I will guide to achieve the same as I did. So first let us see what we are thriving for.
Ta-Da !!!
Doesn’t that look great? Yes, it does.I know that.
So, let’s get started without much a due.
But before that i would like to warn you that this blog might look lengthy as I have included as many images and gifs as I can. I also have included the all the code you need for this blog and have explained everything as easily as I could, so as to make it really easy to follow. So don’t get afraid by the length of the blog.
So, lets break the whole project into small tits-&-bits (yes you got me).
But before that download all the resources needed from here.
Overall Idea
You have already seen the video above and do have a fare idea of what we are going to do…. But still, let me explain you…. Why…again ?? Well, because it was me who wrote this blog and not you. SIMPLE.
So, there are two major parts of the project. First, that is of image processing is taken care by our mighty good old friend OpenCV. Second, that is of visualization (that looks magical), is taken care of by our comparatively new friend ARCore(i.e Augmented Reality SDK by Google).
There are plenty of resources, so if you want to know about both, just do a google-woogle. But still I would be giving you are brief about both of them.
OpenCV is a library of programming functions mainly aimed at real-time computer vision. (Wikipedia…. well i told you to google… right?)
Augmented Reality is a fairly new term. Well, the word “Augmented Reality” is made up of Augmented, i.e. to change and Reality, which of course is reality. So AR is the process of laying computer graphics over the reality so as to give an illusion of the changes reality. Unlike VR, where we are cut off from the original reality and are transported into completely different world which has no connection with reality, in AR the reality remains same and we supposedly add some 3-D content in the environment which appear to be a part of environment itself.
Steps to complete
#1) Project Setup
- Setup a new unity project
- Add Augmented Reality Support via ARCore.
- Adding powerful image processing support via OpenCV.
#2) Project Logic
Project Setup
This step is simpler than you can think. We will just create a new unity project named DoodleAR.
Creating a new project
A quick suggestion from my side to all the Unity beginners to change your Unity Layout to “Tall” if you are beginner in Unity, so that you can follow my guidelines easily. You can do this by simply following the steps given in the gif below.
In this post, I assume that you have basic knowledge of unity. In unity we use materials to change the look of any given object. Let us confirm this by creating a 3D Quad and by setting its material and the texture property of the material. We can simply add quad by right click in the hierarchy panel and then 3D Objects -> Quad. This will add a quad in your Scene Window. Double click on the Quad so as to focus on it in the scene view. Hint: Checkout the gif below.
Now click on the Play button, doing so will open your Game view and you will be able to see the Quad (a white Quad as a default material set to it).
Now let us create a custom material to be applied to the Quad. Go to the assets and right click and choose Create->Material. This will create a material in the assets. Select the material and press “F2" to rename it to “myMaterial”. Now select the quad in the hierarchy window, then in the inspector window(which will be showing the properties of the selected item i.e. quad), select the “Mesh Renderer” and expand the Material.Now drag (don’t click) and drop the myMaterial from the Assets into the Element 0 of the material property of Mesh Renderer. Hint: Checkout the cool gif below.
For this project we need a transparent type of material. So lets change the type of our material. Select myMaterial and change the shader to Unlit-> Transparent from the Inspector Window. Hint: Now already know what the hint is!!! Go check it out.
Now let us assign the texture of the material. For that we will use an image. Import any image from you computer into the assets.You can do this by going to Asset->Import NewAsset and then select any image or by simply drag the image from its location into the Assets window. This will add an image into the Assets and it will be shown in the Asset window. Then, we will apply the image as a texture to our myMaterial. For this click on the myMaterial and drag and drop the image to the Texture area of myMaterial.
Great job !!! We have completed a checkpoint. Now save your scene by pressing Cntrl+S.
In this checkpoint we have created our new project and learned that the look of the object is controlled by the material.
Well that was easy. Give a pat to to yourself.
Adding ARCore Support
We will be using ARCore version 1.9, you can maybe use the latest one available. We will import the Google ARCore Package by clicking Assets->Import Package-> Custom Package and then choosing the ARCore package.Next, a window will appear then we click on “Import” so as to import the package into our project. After the package has been successfully imported, it can be seen in the Assets window.
Now will delete the MainCamera from the Hierarchy as we will no longer need it, as we will be using ARCamera. Go to Assets Window->Google ARCore->Prefabs and drag and drop AR Core Device.
Then we will set the Focus mode of ARCamera to Auto, so that it can focus automatically. To do this, click on ARCore Device in the Hierarchy Window and in the Inspector Window, double click on “DefaultSessionConfig”. Then change the “Camera Focus Mode” to “Auto”.
Now, select the Quad in the Hierarchy and from the Inspector Window change its Transform->Postion to (0,0,0.5) and Transform->Scale to (0.1,0.1,0.1). You might want to double click on the Quad in the Hierarchy so as to Focus on it, as it will disappear from screen because we changed it size to very small. We did this as in Unity 1 unit = 1 meter, so if we keep the scale of our Quad to (1,1,1), then it will be very large in the real world. So, we changed its scale to (0.1,0.1,0.1).
Now, save the scene and build it according to build settings of ARCore.
Steps to Build (Checkout the gif to see all the build steps in action)
#1) Go to File > Build Settings to open the Build Settings window. Under Platform select Android and click on Switch Platform.
#2) In the Build Settings window, click Player Settings.
#3) In the Inspector window, configure the following player settings:
#4) Set the rotation of the application to Portrait only.
#5) Now save the scene.
#6) Click on File-> Build and Run. A window will open, set the name of the apk(maybe DoodleAR) and save it to a desired location(maybe in Folder named apks on the Desktop, so that you can access it easily).
Note: Make sure your device is connected and set on File Transfer and also that USB Debugging is turned on. As Build and Run will run the apk on the device via the USB cable. Or you can also select File> Build, this will just give you the .apk file at your specified location. Then you can copy the .apk file on your device and then install the apk via device. Choose whatever works best for you.
Build Procedure Gif
When you install and run the app, then you will see your quad in front of you.
This means ARCore is working great.
Adding OpenCV Support
#1) Go to Window-> Asset Store and search for “OpenCV Plus”.
#2)Download OpenCV + Unity and Import it also. After it is imported correctly, it will be shown in Assets.
Now, click on Game View Window and click on Free Aspect and then click on the “+” resent at the bottom of the list.
A new window appears to create a new view. Put the name as “DoodleAR” in front of the “Label” and put width and height as 800 and 1280.(Note it is not necessary to match actual resolution of your smartphone.)
Now we will import a custom UI package, which simple adds a few button and a raw image. To add, go to Assets>Import Package>Custom Package. Select the “StampUI.unitypackage” from the Resources folder.
Doing so will add a CanvasUI prefab in your Assets. Now we can again go to the Scene View.
Now drag and drop the CanvasUI into the Hierarchy window. Doing so will add the UI in the Scene.
Double Click on the CanvasUI in the Hierarchy window so that you can view it easily in the Scene View.
The image below explain the role of the UI.
The Viewer is used to see the captured image or the processed image.
Capture Button will be used to capture the screen.
Color Button will be used to change the Color of the Doodle(once we have captured them.)
Stamp Button will be used to put the captured and colored doodle in the world space using AR.
Now we will apply the our custom material i.e. myMaterial to the RawImage Component which is a child of Canvas. So, click on Canvas to see its children and then click on RawImage. Now drag myMaterial from the asset to the material property of the RawImage in the inspector window.
Now we will convert our Quad into a “Prefab” as it will be only be used later. Now what is a prefab ???
Prefab Assets act as templates. You create Prefab Assets in the Editor, and they are saved as an Asset in the Project window.
This can be easily done by dragging the Quad from the Hierarchy Window to Assets Window.
Those who are thinking that it is just another drag and drop tutorial, let me clear your doubts…..No it’s not a drag and drop tutorial. You don’t believe me. Okay com’on lets code. Now let us create a script which contain all the logic of our application. For doing so, right click in the Hierarchy Window and click on “Create Empty”, this will create an empty game object and we can add the script on this game object. Rename the freshly created “GameObject” to “DoodleObject”.
Click on the “DoodleObject” and then in the Inspector Window, click on the “Add Component” Button and create a new script. Name your script as “DoodleARScript”.
Doing so will add a script named “DoodleARScript” on “DoodleObject”. Now let us put some logic inside our script.
Till now it was just preparation (it was….now is the easy part…and well you have already completed 39.69% of the project…Yup, not 40 % only 39.69%). Now embrace yourself for the actual logic that we are going to use.
Project Logic and Road-map
Step 1- Capture an image.
Step 2 -Convert the image into Black and White using OpenCV (known as Binarization)
Step 3- Add an Alpha Channel and set transparency of all the white pixels to 0 and set alpha of all the black pixel to 255. So as to create a transparent effect.
Step 4- Add color to image.
Step 5- Apply texture to Quad and anchor the Quad in AR.
Step 6- Feel a great sense of achievement.
Now open up your DoodleARScript by double clicking on it. You will see an empty script just like this. It will only contain 2 function i.e. Start() and Update().
Note:
I will be using //Add blocks which will indicate the code you need to add in the script in each step.
Step 1 : Capture an image
Once you have added the code to your created script, save the script and go back to Unity Editor. You will find out that there two fields under your DoodleARScript. Assign Canvas to Canvas and RawImage to Preview.
Now we will add an “onClick()” listener on the “Capture” Button present in the canvas.This will execute a function defined by us whenever the capture button is clicked by the user. Basically, we need to provide a GameObject and then we will be able to access the public function in the scripts assigned to the GameObject (DoodleObject in our case). We will do this by selecting the Capture Button under the Canvas and then in the Inspector we will find out the component named “Button (Script)”. Under this, we will click on the small “+” Button. This will show you an empty object and “No Function”. Now drag the DoodleObject from the Hierarchy Window into the GameObject field. Then simply click No Function, and a drop down will appear, then DoodleARScript->StartCV().
Hint:Watch the gif given below to see it in action.
Well if you are not able to see the “StartCV()” function listed in the “DoodleARScript” then you might have made a silly mistake even after receiving a warning from me. Check if the “StartCV()” is “public” or not.Make sure your StartCV() is public.
Now is the time to see what we have done till now in action. Simply save your scene and File->Build and Run your project. Till the project is building up; you can also build up your excitement as this will one of our checkpoint.
After you have built your app and it boots up in your device, you should be able to click on “Capture” Button to capture your screen and the captured image will be showed in the RawImage View. Somewhat like shown below.
If something doesn’t work as it should do checkout the last steps and see that you have done everything correctly.
And well if it does work correctly, then Hurray !!! Give yourself a pat, go and have a break, maybe a drink or so (only soft drinks allowed… “OldMonk” being an exception) and well go to pee(it’s important). And do tell me in comments what did you do after this checkpoint.
CheckPoint_1 completed !!! It’s party time or maybe pee time.
Step 2 -Convert the image into Black and White using OpenCV
Now that we have captured our image, so our next task is to convert this into a black and white image. This is known as Binarization and for this we will take help of our good old friend “OpenCV”.
So there are somethings you need to know before starting this step. Not to worry as you don’t need to know anything in detail.
First is “Mat”, ya Mat, well that’s not the name of my uncle. In simple words we can say Mat is used to represent images in OpenCV. Well all those OpenCV geeks out….don’t freak out…it was just a very simple explanation. For all those who want to know more about Mat in OpenCV, check this link out : https://docs.opencv.org/3.1.0/d6/d6d/tutorial_mat_the_basic_image_container.html
Second is Thresholding.
Image thresholding is a simple, yet effective, way of partitioning an image into a foreground and background. This image analysis technique is a type of image segmentation that isolates objects by converting grayscale images into binary images.
The simplest Binarization methods replace each pixel in an image with a black pixel if the image intensity is less than some fixed constant T or a white pixel if the image intensity is greater than that constant.
Now that’s all that we need to know, let us do some scripting.
Remember, you only gotta add the code, that is inside the //Add blocks, to your previously made DoodleARScript.cs .
Well ….. well ….well!!! We competed our Second CheckPoint. BooYeah !!! Now its time see some Binaraization. So save your progress and like always File-> Build and Run and feel the magic of Binaraization. After you have had fun with the app and have enjoyed the sense of great achievement, go back to script and take a look at how we have achieved this.
We really didn’t do much, we simply created 2 Mats. Converted our texture into Mat (named bgraMat) then converted it into grayScale Mat (named binMat). After that we applied thresholding to binMat and saved the result to binMat. After this we converted the grayScale type Mat (binMat)to BGRA type Mat(bgraMat). Then using the setColor() we passed the color info of bgraMat into texture. SIMPLE !!!
CheckPoint 2 Completed. Not a party time as checkpoint 2 was easy and also as we know that you can’t handle Old Monk well.
Step 3- Add transparency by to your doodleTexture
Time to move on to Step 3 and well that is…
Adding a magical fee to our image by making all our white pixels invisible.
This is not that hard as you are thinking. Once you know the secrets of color World you will be able to do it pretty easily. So there are a lot types of representation of colors like HSL, RGB, #ColorCode, etc. All these represent colors with the help of different properties. You can google them out to know more about them. Moving on….
One such representation is “RGBA” i.e. Red Blue Green Alpha. These four properties are taken into consideration for representing a color in RGBA representation, where Alpha controls the transparency of the color.
So, in this step we will simply find all our white pixels and set their Alpha to 0 and set the Alpha of all the black pixels to 1. Well now you will say, what about all the other colored pixels??? In response to that I would like to advise you to not to drink any more Old Monk. Say What…..??? Ahh….!!! We will only have Black and White pixels as we have already converted our image into Black and White with the help of Binarization in the previous step. Remember ???
Now, let us script our way out of step 3 and complete out checkpoint 3.
For all those, who were not able to understand, what is being done in //Add Block 2, here comes the explanation.
GrayScale Type Image and BGRA Type Images are stored in 1-Dimentional Array named binPtr and bgraPtr respectively.
So, binPtr points to an 1-D array of GrayScale Type Image in which each pixel is represented by only one unit as it has only two possible value (Black = 0 or White = 255).So, size of binPtr will total number of pixels i.e. width * height (let us consider n = width * height ).
Whereas, bgraPtr points to an 1-D array of BGRA Type Image in which each pixel is made up of 4 units, each unit corresponding to B-G-R-A receptively. So, size of bgraPtr will be total number of pixels * 4 as each pixel considers of 4 sub parts i.e. width * height * 4 ( = 4*n).
So we are using a for loop to iterate over both the 1-Dimentinal Arrays (i.e. binPtr and bgraPtr). Note that the maximum value of i in for loop is n (width*heigth), but maximum value of bgraPtr is 4*n(width*height*4).So, how will we iterate over both these array in this for loop?? The answer is using using bgraPos which is equal to 4*i.
Note:
bgraPtr[0*i] = Red Value of the ith Pixel
bgraPtr[1*i] = Green Value of the ith Pixel
bgraPtr[2*i] = Blue Value of the ith Pixel
bgraPtr[3*i] = Alpha Value of the ith Pixel
Let us see a few iteration to get it right.
For i = 0
- binPtr[0] will point to the 0th index of the binPtr array which represents 0th pixel in the GrayScale Type Image.
- bgraPos will be equal to 0*4 = 0. Implies bgraPtr[bgraPos + 3] will equal to bgraPtr[0+3] = bgraPtr[3] which is the Alpha value of the 0th pixel in BGRA Image. Refer the above “Note” with i = 0.
For i = 1
— binPtr[1] will point to the 1st index of the binPtr array which represents 1st pixel in the GrayScale Type Image.
— bgraPos will be equal to 1*4 = 4. Implies bgraPtr[bgraPos + 3] will equal to bgraPtr[4+3] = bgraPtr[7] which is the Alpha value of the 1st pixel in BGRA Image.Refer the above “Note” with i = 1.
For i = 2
— binPtr[2] will point to the 2nd index of the binPtr array which represents 2nd pixel in the GrayScale Type Image.
— bgraPos will be equal to 2*4 = 8. Implies bgraPtr[bgraPos + 3] will equal to bgraPtr[8+3] = bgraPtr[11] which is the Alpha value of the 2nd pixel in BGRA Image. Refer the above “Note” with i = 2.
And so on for all the value of i.
I hope you were able to understand 20% of what you just read. Read this explanation again for 5 times and you will get it.
Now if you have understood the above explanation, now it’s easy. In for loop we simply check the value of binPtr[i] for each i and if it is 255, i.e the ith pixel in the GrayScale Type Image is White, then we change the alpha value of the ith pixel of the BGRA Type Image to 0 by bgraPtr[bgraPos + 3] = 0 i .e. we make is transparent.
Woah!!! That was tricky.
But well we are through it and let us see what we have achieved. Above all, was it worth the try.So, as always save your project and File->Build and Run. Open your app in your device and take a click using the Capture Button and be amazed with what you, yes you have achieved.
CheckPoint 3 Completed. Go have some cookies, because you deserve them.
Step 4- Add color to image.
In this step we will make our Color Button functional. We will add functionality to our Color Button, so that whenever it is clicked it can color our image with random color. So, lets get going.
Com’on let do some scripting.
Now save this script.Now, in Unity, add an “onClick()” listener to the “Color Button”. Now add our DoodleObject to it and choose the Change Color() function under the DoodleARScript.(Just like we did for the Capture Button). The gif below show how it is to be done.
In this script we simply defined an array for colors(//Add Block #1).On the click of Color Button, we choose a color out of our Colors array(//Add Block #2) and apply if to our texture(//Add Block #3).
Noice !!! Seems like we have added some color in our lives. Let us see how works. So, Like always simply save your scene and File->Build and Run.(Make sure your device is connected and File Transfer is selected and USB Debugging is turned ON) The app will be built and will open up in your device once it has completed building up. Now simply click on Capture to capture your screen and then click on Color Button.When you click on the Color Button the color of the texture will change. (Like shown below)
Step 5- Apply texture to Quad and anchor the quad in AR.
Next and the last step is to put the doodle created till now into realm of AR with the help of GoogleARCore. And in this step we will also do some cleaning.
So do as shown in the gif below.(Watch it completely as this is a lengthy one.) Basically we are making the size of the RawImage as the size of the screen. And we are also setting it’s enable to false i..e. it would not be visible when our app starts.
Don’t worry we will enable it using our code. We will enable it at the moment when we click Capture Button(in the //Add Block 2 of the script that we are just going to write.). Doing so will give our app a better user experience. Well believe me, you will see.
Note: Don’t get confused by “GameObject” in the gif given below.It is the same DoodleObject (It’s just that at the time of gif it was not renamed to DoodleObject).
After following this save your scene. Just save don’t Build and Run.
Now we need to get acquainted with the concept of Anchor in GoogleARCore and concept of Pose in Unity.
Pose is used to store both Position and Rotation. Basically it is a combination of Position and Rotation packed as one unit.
Anchors are used to pin objects to the real word in GoogleARCore. They are constantly tracked by GoogleARCore for both their position and rotation and hence remain pinned to the Real World.
So in this step we will simply create an anchor using ARCore. Then we will instantiate our prefabQuad and apply the awesome looking texture, we have created till now,to it.Then we will set the prefabQuad as a child of Anchor so that it remains in its position and is pinned to the real world.
So let us begin our script.
In this script the major thing that we have added is the PutObject() function which does its magic with the help of GoogleARCore and keeps our doodles pinned in the real world. I have added comments in the function which are enough to understand how we are doing do. To understand the stuff properly you must checkout all the function used inside the PutObject function. You can definition to all those function in unity official documentation given in these links RayCasting, Pose, Anchor, ScreenPointToRay etc.
Voila ! Voila ! Voila!
We are done with our project. Let us save and File->Build and Run our project, have a look at what our persistence and hard work has resulted in.
Well, till the time your project is building up you should read Step #6, as it is the most important one.
Step 6- Feel a great sense of achievement.
If you have managed to reach this section, well then clap for yourself and give yourself a pat on your back. I mean, good work, like seriously. If you like the blog then do give me Claps, as this will keep me boosted and I would write more often.
Feel free to share your views about the blog, in the comments below. Well this was my first blog ever, so yeah, ignore my wrong doings,if any. You may ask any of your doubts in the comments as well.
Suggest me some topics on which you want me to write a blog on. What would you guys like to learn???
Feel free to connect with me on Twitter, LinkedIn.
Kudos to everybody
-Jking
(Jatin Pawar)
AR Enthusiast
