Ah, Earth. I’m kinda bored of this place, to be honest. If only I could esca-
Oh, wait. I can. In fact, I could go to the moon, visit Saturn’s rings (while avoiding death, at that), or even travel to another galaxy in a literal second. And you could too, using the wonderous world of virtual reality.
By definition, virtual reality (VR) is a computer-generated (completely digital) simulation of a three-dimensional image/environment. With VR, it feels like you’re immersed in a 3D reality, whether it be similar to this world or an entirely new one, altogether!
Most of us have likely seen it before. Virtual reality (using headsets) looks something like this. ↴
And it can also look like this. ↴
Commonly, consumers use VRs for gaming or entertainment. But outside of personal enjoyment, there are a countless number of industries that virtual reality applies to — ranging from education and medicine to architecture and military.
Now, pause. Wait a minute — it’s time for AR to join the party. Let’s introduce virtual reality’s best friend: augmented reality.
Augmented reality is, essentially, this. ↴
With AR, we’re able to see our real-life environment with a digital augmentation overlay. Virtual objects are basically put into the real world, in real-time. It’s crazy to think that over 300 million people worldwide regularly engage with it via the social app, Snapchat. That’s right — Snapchat filters are an example of augmented reality.
(PSA: Next time you want to impress your friends, just say, “Yeah, I use augmented reality on a daily basis.” They don’t need to know the specifics :))
Aside from the entertainment aspect, there are also numerous real-world uses for AR. This includes travel, locators, translation, and even education (plus many, many more).
In this article, we’ll discuss the types of VR and AR, as well as how they work and what fields the technology is currently being applied to. Let’s take a deeper look into this groundbreaking field.
- Types of VR
- How Does VR Work?
- Types of AR
- How Does AR Work?
- Real-World Applications
Types of VR
Virtual reality systems make it so users are able to be participants in a computer-created reality. That being said, while VR is broadly identified as three-dimensional, not all virtual worlds are. In fact, first-person, second-person, and third-person interactions in virtual worlds are all possible. Allow me to introduce you to the three major types of VR, along with other subsets.
Non-Immersive
Remember when I mentioned this as being virtual reality?
Truthfully, it is. This is an example of non-immersive reality. Most don’t realize that technology like this — Xbox, Nintendo, and so much more — is actually a commonly-used type of VR. As the name suggests, users are still provided with a computer-generated environment, but aren’t fully immersed in it. You are still aware of your surroundings and physical space. Think; a phone game.
So, yes, you practically use VR every day. Ah, technology.
Semi-Immersive
This machine, right below, is another type of VR called semi-immersive.
Hence the name, yet again, a semi-immersive reality is a partially virtual environment. Users remain aware and connected to their physical surroundings, but 3D graphics (or vertical reality depth) are used to create a sense of realism. This type of VR is mainly used for training or educational purposes, with high-res displays and powerful projectors/computers.
For example, in the image above, a driving simulator is shown where the user would feel like they’re driving, although they aren’t on a physical road. The screen displays a virtual street environment, but the user would still be aware that they’re here, on Earth.
Fully-Immersive
Finally, the most commonly known type of VR: fully-immersive. When thinking of virtual reality, we automatically imagine a head-mounted display (the headset) and position tracking devices that can sense when we move our heads around. With immersive reality, users undergo a full first-person experience. It’s as if you are literally there. Sometimes, a treadmill is used to simulate how you’d be walking through that reality. That’s how you could “walk on Mars”.
The display splits between the user’s eyes, which creates a 3D effect. This is the most realistic simulation, using sight, sound, and sometimes touch to interact with the world.
Through the Window
Apart from the three broad types of virtual reality, there are more specific subsets to go into. A through the window system, also known as a “desktop VR”, is where users see a three-dimensional world through a computer screen window.
They then navigate through that window using a control device (i.e. a mouse). Similar to that of immersive reality, this creates a first-person experience.
Mirror World
This guy is standing in front of a giant screen with a video camera, and images from his movements/positions are displayed on said screen. This is a mirror world — one created by Myron Krueger (known as VIDEOPLACE).
Mirror worlds, or projected realities, create a second-person experience in which viewers stand outside of this imaginary world, but interact/communicate with characters or objects within it. Video cameras are used as input devices, and users can see their images merged with a virtual world. This is presented on either a video projected image or a large video monitor. A digitizer is used for the computer to process the users’ images and extract information (i.e. movements, positions, fingers raised).
How Does VR Work?
Okay, so now we have a good understanding of different types of virtual reality. How exactly does it work? Say, we do wanna travel to another galaxy in a literal second. This would require fully-immersive virtual reality for the entire, realistic experience. In order to do so, we need a headset — the thick, goggle-like device that goes over your eyes.
VR Headsets
Headsets, such as the Oculus Rift, are oftentimes referred to as HMDs, or head-mounted displays. Their goal is to make it so wherever you look, the screen on your face follows for a realistic 3D experience. They essentially trick the brain into blurring the lines between what’s digital and what’s real.
There are various types of headsets, but most cheaper ones allow you to clip your phone to the front (like this ↴).
Headsets that are more expensive connect to computers that run the games/apps. Video is sent to the headset through an HDMI cable in the case. Some are wireless, like those from the company, Qualcomm.
Hand controllers may come with the headset, which translates the user’s real-world gestures into the application/game being interacted with. Gaming joypads may also be used.
Headsets use two feeds sent to one display, or two LCD displays (aka liquid-crystal displays, which are flat panels that are used in computer monitors or TVs) with one per eye. Lenses are put in-between the user’s eyes and the pixels, and distance can be adjusted depending on a person’s sight.
Basically, the video/image we see with VR screens is split into two, with one for each eye for more realism.
The headset lenses distort the two images into one 3D image, mimicking the way our eyes would see the world. For example, place a finger in front of your face. Now, close an eye. Open that one, and close the other. Notice how your finger seems to move depending on which eye you’re seeing from. But when both of your eyes are open, your finger seems perfectly centered. This is how the lenses work.
Most headsets use a 100–110 degree field of view, simply because a full 360 would be expensive and unnecessary. 60 frames per second is the minimum frame rate used, since any less would cause a user to feel sick. Although current high-end headsets go much farther than this, with 90fps with Oculus and 120fps with PlayStation VR.
Head Tracking
Some headsets have sensors in them in order to track head movement. This means as you turn, the headset and virtual world follow. Embedded in the headset is a concept called six degrees of freedom, or 6DoF. The system plots your literal head onto an XYZ plane and measured head movement (side to side, forward, backward, etc).
Gyroscopes, accelerometers, and magnetometers make the 6DoF work. Gyroscopes measure/maintain orientation and angular velocity. Accelerometers measure acceleration forces that act on an object to determine its position and movement. Finally, magnetometers measure magnetic forces.
That’s what’s naturally used, unless you’re Sony — whose PlayStation VR uses 9 LEDs around the headset for 360 tracking due to external cameras monitoring the signals.
Headphones may also be used with the headset for a deeper sense of immersion. It could be 3D audio, which developers take advantage of since they could make it seem like a sound is behind, beside, or in the distance in relation to you.
Motion Tracking
This field is a work in progress, but it’s happenin’, alright. When you look down with your headset on, you kinda wanna see hands ‘cause… you have hands. Oculus Touch is a set of wireless controllers that actually make it feel like you’re using your real hands while in a virtual world.
There are buttons and thumbsticks that you click on during the game, which create the desired effect. You could squeeze a trigger to shoot something, or wave/point, which the controller senses. All this for a realistic experience in a virtual world. Ah, doesn’t Mars sound even nicer, now?
So we’ve talked about virtual reality, its numerous types, and how it works. I think AR’s feelin’ a little lonely in the corner now.
Let’s discuss how augmented reality works.
Types of AR
When it comes to AR, there are many different types and uses out there. It’s difficult to get one clear answer on what the main categories of augmented reality are. For broader categories, I’ll be sharing marker-based and marker-less AR (which are literally what they sound like).
Marker-Based AR
With marker-based augmented reality, a static image is required (a trigger photo) so a user is able to scan this and trigger virtual content.
Take this MABU Christmas Card, as an example (L).
The word “markers” describes specific patterns that cameras are able to identify and process. These are typically unique from the surrounding environment. A software, an app, will allow users to scan markers from their device. This may then prompt an animation, text, object, etc. to appear on the screen. Like in VR, tracking also aids in this aspect to ensure that you can move the camera without stopping/distorting augmented effects.
Marker-based AR can be either cloud-based or local-based. If local-based, this means that marker databases can be stored on a device, while that same device does the recognition process. If cloud-based, the databases are stored on a cloud. In this case, recognition would occur on a server and the phone simply sends point clouds to that server. Local-based is quicker, more immediate, while cloud-based takes a few seconds before the augmented reality actually shows.
There are both positives and negatives to this type of AR (of course, nothing’s perfect). On the plus side, it’s very easy to use and the tracking is stable to avoid shaking. However, as the phone camera moves away from the marker, the augmented reality effects will disappear and users will have to rescan. Additionally, lighting has to be optimal since scanning won’t work if the markers reflect certain lights.
Nevertheless, this is still an excellent form of the technology.
Markerless AR
As you can see, the chair in the photo below was not there before the user used an AR system to make it *appear*. Notice how in this method, there’s no target image.
Hence the name, once again, this method doesn’t use markers. Instead, the surrounding environment is scanned with no need for a trigger photo. Oftentimes, for augmented reality effects using this method, it’s better to aim the camera at a table or floor for the virtual object to ‘stand’ on. Although, white backgrounds may make it harder for this to work.
On the plus side, this type of AR is very flexible. However, one negative side to this is that the computer vision only works with textured surfaces (which is why a white background may not be the best option).
Again, nonetheless, this type of augmented reality is super impactful and has thousands of possible uses! Just like in the photo above, it’s great for testing whether a certain piece of furniture would look good in your house — without having to buy it.
How Does AR Work?
So, in short, AR takes the existing environment and uses a digital overlay to place new information. A computer vision will understand the environment around the user, so it can show that digital overlay realistically. The process of making the content look like it’s a part of the real world is called rendering.
Augmented reality happens live, which makes it so that rendering happens each time a new frame comes from the camera.
They literally do this in 30 milliseconds. 30. Milliseconds. Ah, technology.
Furthering in on computer vision, a branch of computer science, both 3D geometry and semantics are used for AR to function. Geometry deals with the, well, geometry of the environment. This implies where objects are facing and where they are in relation to the 3D world. Semantics, the study of meaning, deals with identifying what an object/the environment is. Computer vision processes the distinction between 3D and 2D, so the content will be properly inserted.
Once the AR system recognizes the rendering module, the desired content/effects are displayed. Again, this all happens live.
Real-World Applications
Now, to tie it all together. Although we do use VR and AR commonly for entertainment, there is a wide variety of usages in the real world.
- Healthcare. Using AR, smart contact lenses are capable of being designed — which block out radiation that may harm the eye and risk cancer. With VR, healthcare professionals have a new method of practice to prepare for certain operations. Additionally, many experiments have used Virtual Reality Exposure Therapy to treat mental health issues, including anxiety and PTSD.
- Education. With AR, students (especially younger ones) are more engaged with interactions. Many apps take advantage of this and create educational content that incorporates AR. For example, Quiver is an app where students can color in 3D and bring images to life on the screen. For elementary schoolers, this will definitely be useful. Using VR, students are capable of immersing themselves in the lesson — whether it be visiting a certain time-period simulation or touring a new planet. +1 on the engagement!
- Personal Wellbeing (VR). Oftentimes, people need moments away from reality. A calm, meditative place. Fortunately, VR comes in a clutch with meditation apps and calm spaces. In fact, Guided Meditation VR is a popular VR application that displays calming images, 360, while guiding users through a soothing meditation.
- Translation (AR). Imagine you’re in a foreign country, and you don’t speak the language. Fortunately, using AR, you could simply pull out your phone and place a camera over the words you’re unfamiliar with. And bam live translation. In fact, Google Translate’s app currently has this available!
- And many, many more. The possibilities are truly endless.
Final Thoughts
VR/AR has disrupted the world, and will be our future. There are so, so many potential ways we can use this technology, as well as improve on it as we go. Soon, people will look back to these days, thinking, “Y’all put giant headsets on your faces? You had to use phones to identify buildings? Wow…”
So, pack your bags — or… don’t. You won’t really need ’em. With virtual reality, you can go anywhere, anytime. And thanks to augmented reality, you’ll know exactly where you are.
Key Takeaways
- Virtual reality is a computer-generated (completely digital) simulation of a three-dimensional image/environment. With VR, it feels like you’re immersed in a 3D reality, whether it be similar to this world or an entirely new one, altogether.
- Non-immersive, semi-immersive, fully-immersive, through the window (desktop VR), and mirror world are all types of virtual reality.
- AR takes the existing environment and uses a digital overlay to place new information. A computer vision will understand the environment around the user, so it can show that digital overlay realistically.
- Marker-based and markerless are both types of augmented reality.
- VR and AR can be used for a countless number of real-world applications, including healthcare, education, personal-wellbeing, and translation.
A Quick Message
Before you leave, don’t forget to check out my Linkedin! Email me at aliyao2006@gmail.com with further questions or comments — I’d love to hear from you!
Resources
Check out the resources I used throughout this article!
