An Overview on Mixed Reality

Elena Camuffo, extract from BSc Thesis, Sep 2019

Elle97
8 min readDec 17, 2022
Visual differences between VR, AR, and MR [3].

Historical Background

The Mixed Reality (MR) concept draws its origins in the Early Sixties, despite the widespread development and consistent diffusion across the world in the recent last few years. MR was derived both conceptually and historically from Virtual Reality (VR), which was ideated by Ivan Sutherland in the early sixties. His idea aimed at creating a sort of completely “Artificial Reality”, which surrounds the users and interacts with them.

However, the first example of a real MR application was “Sensorama”, a video arcade invented by Morton Heiling in 1962, which could emanate odors and transmit vibrations to the user. Although, it was not interactive yet. Afterward, Ivan Sutherland continued Heiling’s work at the Utah University, and in 1968 developed the first VR Headset. Then in the 1970s thanks to the diffusion of Computer Graphics in simulations and the improvements of the computing architectures, MR technologies started upgrading to reach the spread and fame they have today.

Since its very beginning, the success and diffusion of MR applications could be easily forecasted since they proved to be well-suited for many different applications, in multiple areas.

Definition and Scopes

The most popular definition of Augmented Reality was proposed by Ronald T. Azuma in his 1997 survey paper. According to him, AR systems must have the following three characteristics:

  1. they combine real and virtual;
  2. they must be interactive in real-time;
  3. they are registered in 3D.

This definition is sufficiently general to be applied to a large number of devices, including Head-Mounted Displays (HMD) and common smartphones. But then, what is the real difference between Augmented Reality, Mixed Reality, and other related similar technologies?

Mixed Reality Continuum

Broadly speaking, it is possible to refer to technologies that merge real and virtual worlds, using the term Mixed Reality. As we can perceive synthetic scenes on various levels of virtualization, it is possible to highlight the peculiarities among different MR technologies, referring to the quantity of real and virtual elements present in the scene. According to Paul Milgram’s paper of 1994, we can define a taxonomy of the various ways in which “virtual” and “real” aspects of MR environments can be arranged.

Reality-Virtuality continuum [2].

To realize this purpose, the concept of the “Reality-Virtuality continuum” has to be introduced. It relates to the mixture of classes of objects presented in any particular setting where real environments are placed at one end and virtual environments at the opposite end.

The former (on the left), refers to environments consisting solely of real objects and displays the processing results using standard displays. The latter (on the right), defines environments consisting solely of virtual objects, e.g., a conventional computer graphic simulation.

Augmented Reality

The definition of “Augmented Reality” given previously, is the most general one and can be associated with the whole continuum spectrum. It can be considered very close to Mixed Reality since they both include the real environment and computer-generated images (CGI).

The difference between MR and AR is mostly semantic but deeply critical to some people working in the field. Thus, Augmented Reality and Mixed Reality can be distinct considering the level of integration between real and virtual elements.

An Augmented Reality application superimposes CGI elements over real ones but avoids every kind of interaction. Mixed Reality includes all the elements, virtual and real, homogeneously, since it has an accurate knowledge of the surrounding world and where things are.

Virtual Reality

As the continuum suggests, the VR environment is completely computer-generated, so it has no reference to the real world and involves all of the user’s senses, isolating them from reality. Since the very beginning of virtual reality technologies, several moral and ethical debates have arisen.

Some people sustain that the artificial elements synthesized by VR systems, so realistic, could induce the user to lose the sense of space and time, and even contact with reality entirely.

The technologies cited above do share some basic features but offer totally dissimilar experiences. Also, target applications and related headsets are often different. VR is becoming increasingly popular in computer games industry, while AR and MR are employed in a larger number of sectors, like Medicine, Games, Navigation, Advertisement, and so on. VR visors are several; the most common on commerce are Oculus Rift and HTC Vive, but there exist also visors suitable both for AR and VR.

Ubiquitous Computing

In 1991 Mark Weiser proposed the concept of “Ubiquitous Computing” (ubicomp) in his seminary essay and wrote that

the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.

His work discussed the diffusion of digital technology in our everyday life as a means to support it. This possibility is step by step, reaching such a high level of integration with our everyday lives that nowadays we can not think of them separately anymore.

Recent advances in sensors, embedded microsystems, and wireless communications, have brought to the constantly-increasing diffusion of distributed computing platforms, described by Weiser twenty-eight years ago.

Ubiquitous computing moves processing and communication technology beyond the personal computer to everyday devices enabling access to digital data anytime and anywhere.

An example of such devices are smartphones, driver assistance systems in cars, home automation, and body area networks. Mark Weiser was predicting the concept of the “Internet of things”.

On the other hand, Ubiquitous Computing lies on the opposite of Virtual Reality. In fact, Weiser stated that ubicomp combines “Virtuality” and “Ubiquity” rather than isolating the user apart from the real world as VR does; the real location and place are considered computational inputs.

Therefore “Ubiquity” describes the degree to which information access is independent of being in a fixed place (a terminal). From these considerations, it is possible to rearrange the Milgram’s continuum into a 2D diagram, extended by Weiser’s statements and consequently taking the name of “Milgram-Weiser continuum”.

The Milgram-Weiser chart [1].

MR philosophy

In order to be respectable and standard, Mixed Reality technologies have to be ruled. Particularly, they need to have defined boundaries as regards the ethical level. The first step towards this regulation was in 2004 when Steve Mann introduced the “Code of Ethics on Human Augmentation” in his Keynote Address at Transvision. This code sustains that:

As we augment our bodies and our societies with ever more pervasive and possibly invasive sensing, computation, and communication, there comes a pointwhen we ourselves become these technologies.

In addition, the document defined three fundamental laws, representing a set of guidelines towards a philosophical ideal (like the laws of physics, or like Asimov’s Laws of Robotics2), rather than an enforcement paradigm.

In 2016, John Rousseau proposed three laws to ensure that augmented and virtual technology positively impacts society. Rousseau said that

The future of human consciousness will be a hybrid affair. We will live and work in a ubiquitous computing environment, where physical reality and a pervasive digital layer mix seamlessly according to the logic of software and the richness of highly contextual data. This is mixed reality.

Rousseau, citing Isaac Asimov’s “Laws of Robotics” suggested these three “Laws of Mixed Reality” that will help us shape the discourse and future development of Mixed Reality with an emphasis on preferable outcomes. The laws are aligned to three significant problem areas, covering the individual, society, and economics.

  1. Mixed Reality must enhance our capacity for mindful attention.
  2. Mixed Reality must embody a shared human experience.
  3. Mixed Reality must respect boundaries between commerce and data.

Examples and development

In Industrial applications, Mixed Reality technologies can be used to create an interactive application that evaluates the placement of new machinery or workstations inside an existing manufacturing plant by visualizing the future plant on the real images. The main advantage is the possibility of evaluating in advance the suitability of the installation by visual inspection, determining whether the new tools are or not in conflict with older ones.

In addition, several MR applications have also been conceived in the military fields. Armed forces have played an important role in the creation and development of wearable augmented reality, starting back in 1963 with Bell helicopter which inspired Ivan Sutherland, the father of Computer Graphics (CG).

MR is also widely used in design and advertising, as a tool for design review and pre-evaluation of the ongoing models while the design is still in the development stage. Virtual models that replace real ones could be used to inform customers and the public about new products.

The medical field is so broad that there are dozens of applications; from ensuring mobility to healthcare workers in a hospital or medical office and granting access to

their health records, to allowing a surgeon from a remote hospital across the world to assist and coordinate an operation at a different location. MR also improves the way healthcare professionals are trained and educated.

XR frameworks taxonomy [4].

XR devices

XR stands for “Extended Reality”, and it is defined as:

Technology-mediated experiences that combine virtual and real-world environments and realities. Here the ‘X’ can be seen as a placeholder for V(R), A(R) or M(R), though it also represents an undefined or variable quality/quantity. Most definitions of XR encompass platforms and content where the user can take digital objects into reality, or, conversely, see physical objects as present in a digital scene.

References

[1] D. Schmalstieg and T. Höllerer, Augmented Reality Principles and Practice, A. Weasley, Ed. Pearson, 2016.

[2] T. Strettona, T. Cochraneb, and V. Narayanb, “Exploring mobile mixed reality in healthcare higher education: A systematic review,” Research in Learning Technology, 2018

[3] https://medium.com/startux-net/the-differences-between-vr-ar-mr-27012ea1c5.

[4] B. Yoo, “Webizing mar contents — wxr library,” University presentation, 2019.

[15] R. T. Azuma, “A survey of augmented reality,” Teleoperators and Virtual Environments 6, 1997.

[16] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino, “Augmented reality: A class of displays on the reality-virtuality continuum,” SPIE Vol. 2351, Telemanipulator and Telepresence Technologies, 1994.

[17] P. Milgram and F. Kishino, “A taxonomy of mixed reality visual displays,” IEICE Transactions on Information Systems, Vol E77-D, №12, 1994.

[18] J. Peddie, Augmented Reality Where We Will All Live. Springer, 2017.

[19] M. Weiser, “The computer for the 21st century,” Scientific American Ubicomp Paper after Sci Am editing, 1991.

[20] G. Schall, “Mobile augmented reality for human scale interaction with geospatial models,” 2011.

[21] S. Mann, “Code of ethics on human augmentation,” 2014.

[22] S. Mann, T. Furness, Y. Yuan, J. Iorio, and Z. Wang, “All reality: Virtual, augmented and mixed (x), mediated (x,y), and multimediated reality,” arXiv:1804.08386v1, 2018.

[23] J. Rousseau, “The laws of mixed reality — without the rose-colored glasses,” 2016.

--

--

Elle97

Computer Graphics and Deep Learning enthusiast. PhD in information engineering at University of Padova.