Mixed Reality

From Virtual Reality, Augmented Reality Wiki
(Redirected from Mixed reality)
Jump to: navigation, search


Figure 1. Milgram's virtuality continuum (Image: Hughes et al., 2005)
Figure 2. Mixed Reality and inputs (Image: developer.microsoft.com)
Figure 3. Devices on the mixed reality spectrum (Image: developer.microsoft.com)
Figure 4. Experiences in the mixed reality spectrum (Image: developer.microsoft.com)

Mixed Reality (MR) is the blending of the physical and digital worlds. The term was originally introduced by Paul Milgram and Fumio Kishino in a 1994 paper entitled “A Taxonomy of Mixed Reality Displays.” While the main goal of mixed reality is to combine the best aspects of virtual reality and augmented reality, it can also refer to the entire spectrum of situations between virtual reality and actual reality. Indeed, Milgram and Kishino introduced the concept of the virtuality continuum (Figure 1) in their paper. In this case, mixed reality can include augmented reality, augmented virtuality, and other mixed configurations. Therefore, mixed reality can be used as an independent concept or to classify the spectrum of reality technologies in the virtuality continuum [1] [2] [3].

Mixed reality unlocks possibilities that before were restricted to the imagination and it is the next step in human, computer, and environment interaction. Its development is made possible by advancements in computer vision, graphical processing power, display technology, and input systems [1].

As an independent concept, MR is an overlay of digital content on the real world that is anchored to and interacts with the real environment. An example could be surgeons overlaying ultrasound images on their patient during an operation. It could also be a predominantly virtual space where real world objects or people are dynamically integrated to produce new environments and visualizations. In an MR environment, the user navigates through the real and virtual environments at the same time instead of exploring only a completely virtual world like in virtual reality. The main characteristic of MR is that the digital content and the real world content react to each other in real time [2] [4] [5].

MR as the virtuality continuum concept represents the possible variations and compositions of real and virtual objects. As can be seen in Figure 1, it ranges from a completely real environment on the left to a completely virtual environment on the right [2] [3].

Microsoft’s HoloLens has been associated with mixed reality. According to Ricci et al. (2015), these type of devices “will be able to seamlessly extend people’s cognitive capabilities of perception and reasoning to influence their actions in both the real and augmented world” leading to a “new kind of smart space in which digital, physical, and social layers are strongly intertwined.” [6]

The adoption and success of MR are dependent not only on the advancement of technological capabilities but also on the exploration of creative possibilities that such a medium brings [7].

Human, computer, and environmental inputs

True mixed reality experiences are possible due to a combination of computer processing, human input, and environmental input (Figure 2). Human input can occur through different means that include keyboards, touch, voice, and kinect skeletal tracking, for example, and advancements in sensors and processing are giving rise to a new area of computer input from environments. A user’s position in the world, surfaces and boundaries, ambient lighting, environmental sound, object recognition and location are examples of environmental inputs that can be captured. Using all the different inputs, human and environmental, as well as the computer processing, movement through the physical world can translate into movement in the virtual reality. Also, the boundaries of the physical world can influence the virtual experiences. Environmental inputs are essential because without them the physical and digital realities cannot seamlessly blend [1].

The Mixed Reality spectrum

The virtuality continuum is defined by the physical and digital worlds at polar ends. The continuum can also be called as the mixed reality spectrum. On one side, it is limited by physical reality (the natural world, where nothing is computer generated), and on the other side limited by virtual reality (where everything is computer generated). As one moves from the left of the spectrum to the right, the level of digital augmentation increases. Therefore, the continuum covers all possible variations and compositions of real and virtual objects. [1] [2] [8].

The augmented reality and virtual reality experiences available today represent a small part of the spectrum (Figure 3). Nevertheless, they are subsets of the larger MR spectrum. For example, the majority of mobile phones today have little to no environmental understanding capabilities. The experiences they provide cannot seamlessly blend the physical and digital realities [1].

When video streams of the physical world are overlaid with digital graphics, this is augmented reality. The experiences that occlude the user’s view and present a completely digital world are virtual reality. Between those two, there are other types of experience that are considered mixed reality. For example, starting with the physical world, another person’s avatar is shown where they were standing when leaving notes - an asynchronous collaboration experience [1].

MR, as an independent concept or as referring to the spectrum, attempts to combine the best of virtual reality and augmented reality. When the real and virtual worlds are merged together, new environments and experiences are possible, and physical and digital objects can coexist and interact in real time [2].

Main components

Real environment (natural environment) - It comprises of the natural world - all living and non-living things occurring naturally on Earth. Most virtual environments take the natural environments has models [2].

Augmented Reality - It brings aspects of the virtual world into the real world. In the virtuality spectrum, it is closer to the real environment than the virtual environment. The augmented reality users remain in the real world while experiencing a digital layer of information or graphics overlaid on the real world scene. The digital content is not anchored to the real world. This means that both the real world and the digital content do not interact (i.e. they are not able to respond to each other) [2] [3] [4].

Augmented virtuality - it describes the environments in which real objects are inserted into virtual environments. It is the inverse of augmented reality. For example, using a kitchen remodeling scenario, with augmented virtuality technology a homeowner could see and interact with virtual appliances, experimenting with different layouts in a virtual representation of the real kitchen [2].

Virtual Reality - A completely digital environment which provides the greatest level of immersion to the user. The virtual world may simulate the properties of real world environments or exceed the limits of physical reality. In virtual reality, the total immersion requires stimulation of the user’s senses so that the brain accepts the virtual world as a real environment [2] [3].

Windows Mixed Reality

Microsoft has been talking about implementing MR for years, intending to integrate augmented reality and virtual reality headsets so they can work together in harmony across virtual and physical environments. At Microsoft’s 2017 Build conference, the company presented some mixed reality demos that makes Microsoft’s ambitions achievable, including an experience that brought together HoloLens and the Acer's Windows 10 Headset [9].

There are two main devices that can deliver the Windows 10 mixed reality experiences: holographic devices that have the ability to place digital content in the real world such as the HoloLens, and immersive devices that have the ability to create a sense of “presence”, placing the user in a digital experience like the Acer's Windows 10 mixed reality headset [1]. These devices and their integration with Windows 10 were made possible with the release of the Windows 10 Creators Update, that brought mixed reality capabilities to the operating system [10]. According to Microsoft, Windows 10 mixed reality is built with the entire spectrum in mind, merging digital representations of people, places and things with the real world [1].

Microsoft is in a unique place, being the only company with a single platform that can support augmented reality and virtual reality. It also has a lot of hardware experience that others do not. While the company has the software and hardware capabilities to bring mixed reality to the general public, it also has the challenge to captivate developers and make them believe in the company’s vision for its new platform [9].

Different MR experiences will have different locations in the virtuality spectrum (Figure 4). A fully realized mixed reality experience would stay in the middle of the spectrum, perfectly blending the real and digital worlds. Skype on Microsoft HoloLens is an experience that is further left of the spectrum, while HoloTour provides an immersive experience that pushes the limits of HoloLens’ immersive capabilities and places it on the right of the continuum. These and other experiences are Microsoft’s attempts to explore and carve a niche for itself in the mixed reality field [1].


  1. 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 Microsoft. Mixed Reality. Retrieved from https://developer.microsoft.com/en-us/windows/mixed-reality/mixed_reality
  2. 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 Reality Technologies. Mixed Reality. Retrieved from http://www.realitytechnologies.com/mixed-reality
  3. 3.0 3.1 3.2 3.3 Milgram, P. and Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems, E77-D(12)
  4. 4.0 4.1 Foundry. VR? AR? MR? Sorry, I’m confused. Retrieved from https://www.foundry.com/industries/virtual-reality/vr-mr-ar-confused
  5. Bilinghurst, M. and Kato, H. (1999). Collaborative Mixed Reality. Proceedings of the First International Symposium on Mixed Reality
  6. Ricci, A., Piunti, M., Tummolini, L. and Tummolini C. (2015). The mirror-world: preparing for mixed-reality living. IEEE Pervasive Computing, 14(2): 60-63
  7. Hughes, C.E., Stapleton, C.B., Hughes, D. and Smith, E.M. (2005). Mixed reality in education, entertainment, and training. IEEE Computer Graphics and Applications, 25(6): 24-30
  8. Kirkley, S.E. and Kirkley, J.R. (2004). Creating next generation blended learning environments using mixed reality, video games and simulations. TechTrends, 49(3)
  9. 9.0 9.1 Hardawar, D. (2017). I finally believe in Microsoft's mixed reality vision. Retrieved from https://www.engadget.com/2017/05/12/microsoft-hololens-mixed-reality/
  10. Zelenika, K. (2017). Windows 10 Creators Update with mixed reality launches. Retrieved from https://vrworld.com/2017/04/10/microsoft-windows-10-creators-mixed-reality/