Image taken from Unsplash

Using ARKit To Display Video In Augmented Reality

What we will be doing

As a kid growing up, I was really fascinated by the animated Daily Prophet prints in the Harry Potter series. If you have watched them before, the news articles come up ‘live’ just like what you see below.

In this tutorial, we will be recreating this using ARKit. We will first detect an image of Sirius Black, and then overlay a video to be played on top of the image. Let’s get started!

Tutorial

Before we being this tutorial, let’s gather all the materials we will need for this tutorial to work.

Gathering the resources

I will be using this particular photo

First what I have done is to browse Pinterest to retrieve this particular print. Next, I have cropped the image that we will be detecting and overlaying the actual video.

Cropped image

And finally, we will be playing a video over this detected image.

I will be using this particular video.

Creating a new project

Create a new AR Project

Language: Swift, Content Tech: SceneKit

Now that you have created a project, let’s create a new AR Resource Group in our Assets Folder, Assets.xcassets

Add our cropped image (mine is named black ) and give it a size. The size is our actual cropped image size on our printed photo. From my measurements, my printed photo was around 14x15cm. Once we key in 0.14 meters in width, Xcode automatically generates the height.

Next, drag and drop the video file that we will be using to play over the detected image.

Video file black.mp4 added

Make sure when we add the video, we add it to our Project Target.

Adding in code

Delete all the code in ViewController.swift and replace it with the above. What we have done here is set up the AR Configuration to track images and set what images our app should track.

Next we will use one of the delegate methods to add our video over the detected image. The delegate method informs us as it is about to add a node to the scene. This node will be empty and we can customise is accordingly.

But before we enter new code, this will be the logic flow:

1. First detect when our app has added a new node in an anchor
2. Check if our anchor matches ARImageAnchor , if it matches then it means that our image was detected
3. Create a video node containing our video file and add it to a video scene. Adjust all the positions necessary
4. Create a new plane that has the same orientation and size as our detected image
5. Add our video node to the plane
6. Create a new node containing the plane
7. Add this new node as a child node

This is what we will end up with. Let’s run the app and see the outcome!

Nice! Our AR App is working as expected! However, there is a small issue. The original Daily Prophet news, has a loop effect, so let’s loop our video over the image. ARKit enables you to develop the largest AR platform in the world. So, it is beneficial to learn ARKit from the best ARKit tutorials.

Looping the video

Edit our code to be as above. What we have changed is:

1. We use AVPlayer to play item
2. Initialise our video node using AVPlayer
3. Add an observer such that when the player finishes its current item, reset the video and play again

Go ahead and run the application! The final output should be very similar to what we have set out to achieve!

Conclusion

In this tutorial you have seen how easy it is to get started with ARKit it really makes getting started with AR a fun process! There is a lot more you can do with ARKit, such as adding 3D Models / WebViews over the tracked images or adding new objects / models over detected planes / feature points etc. But for now, we have accomplished something really cool in just few lines of code!

As usual, feel free to find the source code for this project at my repo below:

Bilguun132/Harry-Pokker

If anyone finds these useful, feel free to share this or let me know should there be an error / bad practice / implementations.

Have fun coding!