Markerless inside-out tracking
Contents
Introduction
Positional tracking is an essential component of both virtual reality (VR) and augmented reality (AR), contributing to a greater sense of immersion and presence. It determines the position and orientation of an object within the environment. In VR, this allows for the movements of the user to be translated into the virtual environment, and in AR it is essential for the placement of digital content into real objects or spaces. Markerless inside-out tracking is a composite term that defines a form of positional tracking that uses two specific methods: markerless tracking and inside-out tracking. [1] [2]
Markerless tracking is a method of motion tracking that avoids the use of markers (also known as fiducial markers). These markers are usually placed in the environment or in the head-mounted displays (HMDs), helping the system determine the users or camera position. The Markerless method uses instead natural features already present in the environment, for tracking purposes. [3] [4] This method is considered more flexible and effective since there isn’t the need for a prepared environment with markers. [5] [6] The decreasing cost of computer vision is another factor that has made markerless tracking an attractive alternative of tracking. However, this method still has some problems, such as motion blur and fast motion affecting tracking. [7] It also demands a trade-off between precision and efficiency. [2]
Inside-out is a term that defines the position of the cameras or other sensors relative to the object to be tracked. If they are placed in a stationary location exterior to the HMD, for example, it is considered outside-in tracking. In inside-out, the camera is placed in the HMD. [8] There is great interest in inside-out tracking since it simplifies the setup and allows the user a greater sense of freedom by not being limited to a specific tracking space. Inside-out positional tracking has the potential to make VR completely wireless, more convenient and accessible. Some have gone as far as saying that it was the “holy grail” of the VR scene. [9] [10].
With inside-out, the data that is gathered is generally processed by the headset, giving the user an increased mobility within a room. It can also be the case that the sensors are connected to a computer that is carried by the user or the signal is sent wirelessly to a stationary processing unit, but since inside-out tracking aims to provide more freedom of use the optimal solution is the headset providing all the computational power. [2]
There is a great deal of interest in markerless technology. While marker-based systems are still better for accuracy and latency, there has been big investments in researching and developing markerless tracking for it to become the leading positional tracking technology. Indeed, it became the big VR trend of 2017, with the potential to make VR more convenient and accessible. [4] [7]
High-end VR headsets use external cameras or external markers to calculate the user’s position in space and translate it to the virtual environment, and mobile VR systems have no positional tracking. Markerless technology could not only simplify the setup for high-end headsets but also expand the capabilities of mobile VR. [7]
Companies are starting to offer interesting inside-out systems. Microsoft has been investing in this technology for a while, with the HoloLens and, more recently, with the new line of mixed reality headsets which includes the Acer Windows Mixed Reality Headset (Figure 3). Oculus has announced a markerless prototype called Santa Cruz, and Qualcomm’s 835 reference headset is designed to help other manufacturers build their own systems. Intel has also shown a version of its Project Alloy - a “merged reality” HMD that uses markerless tracking. Finally, Google has presented a markerless system named WorldSense intended for use with the VR headsets for the Daydream platform. [4] [7] [8]
Advantages of markerless inside-out tracking
The benefits of using markerless inside-out are those that come from the individual advantages of each tracking method.
The main advantage of markerless systems is the freedom - the increased mobility - they give to the users. While marker-based systems provide great accuracy, they need external sensors. With markerless, the computer can calculate the headset’s position based on signals like depth and acceleration, without a fixed point of reference. Another advantage is that markerless systems may not suffer from occlusion. This occurs in marker-based tracking, when the user gets out of the field of view of the external cameras. On the other hand, having a device that can determine orientation and position independently of external cameras or sensors provides a deeper sense of immersion. Finally, markerless technology is essential for augmented and mixed reality since these technologies demand more mobility from the systems. [3] [4] [7]
Currently, markerless tracking systems still lack in accuracy and latency, although improvements are constantly made. It requires good computer vision, that is a bit further behind outside-in solutions. Another disadvantage, related to inside-out tracking rather than the markerless aspect, is that all the computational work must be done by the HMD. [4]
Markerless
- Can work within unprepared environment;
- No special hardware required;
- No fiducial markers;
- Able to use natural features to calculate position and orientation. [5] [7] [11]
- Increased mobility and freedom;
- Not affected by occlusion;
- Provides a deeper sense of immersion since the user is not tethered. [9] [12]
Inside-out
- No need for external sensors and cables, but rather requires only a light-emitter;
Project Tango and Project Alloy and the future of inside-out tracking
Tango is a Google’s project that allows smartphones and other mobile devices to understand their location in space. It is a software and hardware solution that offers inside-out technology and markerless tracking. According to Google Tango’s director of engineering, Johnny Lee, “As you can see [inside-out positional tracking] clearly works on this phone…We’ve even had people strap a tablet sized device with Tango built in into a custom VR headset and the positional tracking worked just as well as it does here…The only thing really holding us back right now are the thermals. Right now, phones just get way too hot if we ask them to run positional tracking and split-screen stereoscopic image at 90 fps…As Daydream matures and Tango continues to improve, the sophistication around tracking will also evolve over the next 2-3 years.“ [10]
In 2016, Intel announced Project Alloy (Figure 1) - a VR system. It is also a markerless inside-out tracking device. During CES 2017, Intel showed off a developer kit of the project. Project Alloy uses a tiny camera array called Intel RealSense. These cameras are able to make a 3D map of the environment (e.g. a room) and the objects in it. The headset does not need wires, and it is powered by a battery. [13]
An inside-out system with perfect accuracy is considered the holy grail of positional tracking. While it is expected that the technology will evolve considerably in the future, outside-in will continue to exist as the best solution for high-end VR systems. This specific market will probably be the last to adopt inside-out tracking. The implementation of augmented reality systems will also likely drive the use of inside-out since those systems require great mobility. According to Youssri Helmy, CEO of Eonite, when referring to inside-out said that “It’s got to happen. We think it's crucial. It's like the invention of the mouse with the GUI.” [4] [7]
Markerless, inside-out tracking Devices
Markerless, inside-out tracking Systems
Windows MR Tracking - Microsoft
WorldSense - Google - inside-out tracking system for standalone Daydream headsets
References
- ↑ Boger, Y. (2014). Overview of positional tracking technologies for virtual reality. Retrieved from http://www.roadtovr.com/overview-of-positional-tracking-technologies-virtual-reality/
- ↑ 2.0 2.1 2.2 Ziegler, E. (2010). Real-time markerless tracking of objects on mobile devices. Bachelor Thesis, University of Koblenz and Landau
- ↑ 3.0 3.1 Virtual Reality Society. Virtual reality motion tracking technology has all the moves. Retrieved from https://www.vrs.org.uk/virtual-reality-gear/motion-tracking
- ↑ 4.0 4.1 4.2 4.3 4.4 4.5 Klein, G. (2006). Visual tracking for augmented reality. Ph.D. thesis, University of Cambridge, Department of Engineering
- ↑ 5.0 5.1 Zikas, P., Bachlitzanakis, V., Papaefthymiou, M. and Papagiannakis, G. (2016). A mobile, AR inside-out positional tracking algorithm, (MARIOPOT), suitable for modern, affordable cardboard-style VR HMDs. In Digital Heritage. Progress in cultural heritage: documentation, preservation, and protection. Springer International Publishing, Switzerland
- ↑ Lima, J.P., Roberto, R., Simões, F., Almeida, M., Figueiredo, L., Teixeira, J.M. and Teichrieb, V. (2017). Markerless tracking system for augmented reality in the automotive industry. Expert Systems With Applications, 82: 100-114
- ↑ 7.0 7.1 7.2 7.3 7.4 7.5 7.6 Fang, W., Zheng, L., Deng, H. and Zhang, H. (2017). Real-time motion tracking for mobile augmented/virtual reality using adaptive visual-inertial fusion. Sensors, 17
- ↑ 8.0 8.1 Boger, Y. (2014). Positional tracking: "Outside-in" vs. "Inside-out." Retrieved from https://vrguy.blogspot.pt/2014/08/positional-tracking-outside-in-vs.html
- ↑ 9.0 9.1 Robertson, A. (2017). Self-tracking headsets are 2017’s big VR trend — but they might leave your head spinning. Retrieved from https://www.theverge.com/2017/1/12/14223416/vr-headset-inside-out-tracking-intel-qualcomm-microsoft-ces-2017
- ↑ 10.0 10.1 Durbin, J. (2016). Google: Wireless positional tracking “solved”, but heat still a problem for VR. Retrieved from https://uploadvr.com/inside-out-google-solve-tracking/
- ↑ Hsia, J.S. (2017). Markerless AR: 4 Things developers need to know. Retrieved from https://developer.att.com/blog/markerless-ar-developers
- ↑ Langley, H. (2017). Inside-out v Outside-in: How VR tracking works, and how it's going to change. Retrieved from https://www.wareable.com/vr/inside-out-vs-outside-in-vr-tracking-343
- ↑ Nunez, M. (2017). Intel’s Project Alloy is what a VR system should be. Retrieved from http://gizmodo.com/intels-project-alloy-is-what-a-vr-system-should-be-1790818104