Basics of Augmented Reality

8 min readApr 15, 2020

Augmented reality will be multi-billion dollar industry in few years. AR revenue trends will give you an idea of how fast AR/VR is growing. The applications of AR are limitless. There are tons of resources out there to learn about Augmented Reality, it’s applications and revenue trends. This article is getting you up to speed on AR (if you are new to it) before we move to our next interesting article.

What is Augmented Reality?

As per Wikipedia,

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.

In simple words, AR is superimposing computer-generated virtual objects in the real world. These virtual objects usually are 3D models and we can interact with them in the real world. Example, if you are planning to buy a new sofa and you are confused which one to buy, you can put a 3D model of the sofa into your living room and see how it looks and then choose which one to buy, Ikea Place is one such app which helps you do this.

All the AR magic of rendering virtual objects, getting camera feed, anchoring objects in the real world space, etc. happens in real time. The best part is different people can see the same AR magic from their perspective and at the same time. AR finds its application in tourism, education, industrial training, retail, gaming and advertisement industry.

Types of Augmented Reality

Mixing of real and virtual world can be done in many ways but, broadly there are 2 basic types of AR. Each of these types have their own set of usages.

Marker based AR

Markers are pattern recognized by the AR image recognition algorithm and display the 3D object based on marker. For example, you can scan a QR code (marker) and superimpose a 3D model on top of it.

Markers can be an image, QR code or small object, which are trained. These markers are later recognized by the camera to render the 3D model. The higher the distinctive elements in the image the higher is the chance of displaying an accurate 3D model after scan.

Usage of Marker-based AR is very simple and practical as it just requires a 2D image to scan and render a 3D model. So any existing 2D image should work, take a skeleton image from your biology book, scan it and you can see an actual 3D skeleton standing. If the 3D skeleton is interactive you can see all sorts of details, like the bone name, it’s purpose, movements, etc.

2. Markerless AR

As the name suggests there are no markers to render virtual objects, instead it uses GPS, position, orientation, SLAM to understand the surroundings and the objects in it. Then renders a virtual object in the real world, once placed object will stay in its position even when we move the camera around. This is the most fascinating feature, thus by putting a sofa in your living room you can walk around the room and see how it looks from various angles.

There are subtypes for this category:

2.a. Location based

Virtual objects are kept at specific locations and when the user moves to that location they appear on the screen. Information is gathered from various sources like GPS, compass, accelerometer, orientation and odometry.

These can be used in the gaming industry like Pokémon GO, or the manufacturing industry to augment virtual objects at a specific site or location.

2.b. Projection based

Real world physical objects like wall, floor, table-top, hand, etc will become the screen to project the virtual objects. Understanding the surroundings, locations and depth sensing technologies are used to project virtual objects.

The pieces of this system are a user to interact, a camera to gather information and finally a projector to project the virtual objects in the real world. The user(s) can interact with the projection and based on the interaction rendered virtual object reacts. Projection Based AR can be used to show a dial pad in your palm, fun games, education, advertisement, military, etc.

2.c. Superimposition based

A real-world object is superimposed with the virtual object. Imagine seeing an engine, scan it and suddenly you can see all the moving pieces inside. Not only that you can interact with these pieces to know more about them or see how it all fits together to make the engine, pretty damn cool. The moving pieces here are a real world object detection, rendering a virtual object on top of it and giving interaction to the augmented object. You can check out Lenskart or DeerAR to experience the power of superimposition.

How does AR magic works

Computer Vision plays a vital role in the development of AR technology. Thanks to all the awesome Computer Vision algorithms it’s becoming easy to detect a corner in the real world, or detect a surface horizontal/vertical, or detect an object or it’s position, depth, etc.

Few of the moving pieces of AR are:

1. SLAM (Simultaneous Localization and Mapping)

It’s used in robotics for helping robots to get an understanding of their surroundings, which they use to move around or pick something. Data from sensors are gathered to understand the unknown environment and this data is used to find robots position in the environment.

In odometry for augmented reality, SLAM is the computational problem of constructing or updating a map of an unknown environment while simultaneously keeping track of an agent’s location within it. As mentioned by MAXST, SLAM plays an important role in many areas such as location recognition, location tracking, display, graphics, interface, and interaction.

2. Depth sensor

Depth sensors are used to map the depth of an object in the real world. Using this we get a more immersive experience in AR and a virtual object can be placed behind a real world object.

3. Display

A display screen is required to blend the real world with virtual, be it your smartphone or a projection on the real world. This is the place where all the AR magic is visible and become a source of interaction with the virtual objects.

4. Camera and other sensors

Camera is the main input source to capture video feed for any AR app, these data will be passed on to SLAM and other algorithms to get an understanding of the surroundings. Data from other sensors like magnetic sensors, orientation sensors and GPS are also used to help in giving accurate and immersive Augmented Reality experience.

5. Projection

AR headset will have miniature projectors attached which takes data from algorithm output or other sensors and projects a virtual object on the surface. Use of projection in AR is still in its infancy state

6. Processing

For processing all the data from different sensors and running algorithm AR device needs processing power CPU, GPU, flash memory, RAM, etc.

A mobile device will have the processing power and all necessary sensors. All the big AR players are developing their way of interacting with these sensors and using different algorithms to give you the best AR experience.

There are lots of tools out there for the development of AR experience. A Jelvix blog summarizes the major players on AR space.

Augmented Reality the good and the limitations

The Good:

1. AR opens up a new way of thinking in every industry. In education AR can be helpful for students to understand the laws of physics using visualization. In the manufacturing sector, it can be used for training and debugging issues. In the retail industry it can help in blending extra information about the product along with the product.

2. AR in eCommerce is one of the most practical use cases. You can virtually decorate your whole house and make the judgment of how things look once everything is in place.

3. AR can make anything content rich and interactive with great visualization. Imagine you are looking into a model of a house and you scan the pamphlet and a real 3D interactive model house appears. All this is possible because of AR

4. AR finds itself in the gaming industry as well.

5. Tourism can use AR to show interactive details about the site.

Apart from the usages mentioned above, AR can be used in any industry. AR brings a unique user experience in the table, which will improve the product experience and interaction without using it or buying it.

AR can be a differentiating factor among competitors.

The Limitations:

1. It violates the privacy of the user. To make AR work it has to analyze the camera feed, which can be a privacy breach if used in sensitive areas.

2. AR tech is costly, but thankfully with the integration of advanced sensors in mobile devices, AR is seeing it’s light of the day in Mobile devices. However, most of the cool AR sensors are only available on flagship devices.

3. There are a plethora of AR development toolkits available out there. Sometimes it becomes difficult to choose which one to use. Each one has it’s own benefits and limitations. Based on AR application needs you need to decide on which tech to use. On top of it, development becomes a little challenging if your userbase is distributed in the Android and iOS space.

4. Lack of AR app design and development standards. As a result, compatibility and contribution to the app becomes challenging

5. With the new user experience there is a learning curve for adoption of this technology. We all know how difficult it was for elderly people to adopt smartphones. Explaining the new user experience so that the end user uses it smoothly, should also be a topic of discussion during AR application development.

Conclusion

I think, AR tools are still going through the phase of development and it will take some time for this tech to be mature and widely adopted. Big AR players are investing heavily in making their AR toolkit more mature and adding features so that we will get immersive AR experience. AR application developers are the ones who will be responsible to make these tools popular.

In my next article, I will be talking about one such tool which is still under development, but it is in the process of solving AR compatibility and standardization issue. The tech will in turn help in reducing the cost of AR application development.

And finally, we will see how to develop AR Chess game displayed on top of this article.