Mixed reality: the best of both worlds

Offering businesses the perfect combination of digital innovation & real world success

We’re all familiar with the expression ‘the best of both worlds’. But nowhere is the phrase more apt than when it comes to characterising mixed reality (MR), one of technology’s newest and most talked about innovations.

In MR, real and virtual worlds merge seamlessly to create a new environment where physical and digital objects coexist and, crucially, interact with one another in real time. It is in the worlds of business, medicine and education where this technology’s true disruptive potential can be felt most strongly.

MR. . . not AR or VR

Before I explain why it’s becoming so important, I need to first distinguish MR from both augmented reality (AR) and virtual reality (VR). After all, as its latest figures reveal, IDC still lumps them all together as it predicts that the technologies are set to become a $165bn industry by 2020.

Although clearly interrelated, these three realities offer significant different experiences. While VR shuts out the real world and transports us to an entirely digital environment, AR works by layering 2D or 3D content on top of real objects or actual physical locations, turning the physical into the digital. MR, meanwhile, is a hybrid of the two and more sophisticated than both.

“Believability is a core design principle in mixed reality.”

By scanning our physical surroundings to create a 3D map, MR devices know exactly where, when and how to insert digital content for ultimate precision and realism. This changes things considerably because, unlike AR or VR, users can then interact with their new environment (and vice versa) with extraordinary tangibility in real time. As one of my Microsoft MR engineering colleagues, Cris Derr, said recently, “Believability is a core design principle in mixed reality.”

A disruptive lens

I believe that of the three, it is MR that will drive the industry’s growth. At a time when businesses must choose between either embracing disruptive technologies or being left behind, it is MR that offers a path to true digital transformation.

It is for this reason that, at Microsoft, we’re continuing to innovate our own MR solution, HoloLens . Combining a headset with a suite of supporting Microsoft Windows software, HoloLens works by extending the realities around the user. For customers of all shapes, sizes and sectors, it’s helping change the game in terms of how they operate – from pioneering advances in areas like 3D design and scenario planning, to enhancing productivity and collaboration.

From surgery to safety checks

Our work with University College London (UCL) is a great example. Medical professionals and students are using HoloLens to make the process of studying the human body and treating patients easier and more accurate than ever before. Whereas 2D CT scans can be difficult for doctors and surgeons to interpret in the context of a real 3D lifeform, HoloLens lets them create interactive 3D models of bones, organs, blood vessels, muscles and even entire human bodies. This, in turn, means they can get to grips with their exact composition, focus in on areas of interest, diagnose issues and precisely plan surgical procedures. What’s more, because every project the UCL team does is open source, they are able to share everything they discover with other medical practitioners and students all over the world.

Similarly, engineers at University of Cambridge’s Construction Information Technology (CIT) Laboratory are currently trialling HoloLens as a way for construction inspectors to spot potentially dangerous cracks in bridges without having to physically visit them. Instead of sending a team of safety inspectors thousands of miles, workers at the location can take pictures of the structure using DSLR cameras before uploading them to a cloud service. The safety team then use HoloLens to view the images stitched together as a single 3D model. This lets them zoom in and out of the bridge, rotate the image and ‘walk’ around it, no matter where they are. The benefits in terms of costs saved and the ability to identify and tackle problems quicker are extremely promising and, of course, highly replicable across different types of structures.

Designing the future

While the application of MR in fields like medicine, academia and health and safety have the potential to directly improve quality of life for people and communities all over the world, many private sector organisations are also using MR as a business advantage.

Take Gensler, the world’s largest design and architecture firm and, as designers of the first ever 3D printed office, no strangers to innovation. The firm is using SketchUp Viewer for HoloLens – a MR solution developed by Trimble – to bridge the gap between the on screen world and the real one. This lets the firm’s architects ‘step inside’ holographic representations of their creations to interact with, and evaluate, them in the context of the actual physical space.

The difference this is making to Gensler’s operations is significant. Collaboration between colleagues in separate locations is faster and easier than ever, allowing decisions to be made more confidently and consistently across the entire building process, from design, to construction and even maintenance. What’s more, it also allows employees to share digital assets with clients and other stakeholders in the same natural, immersive way, helping keep all parties informed, in sync and engaged throughout the project.

Business at another level

In the same way that Gensler is using HoloLens to optimise internal work flows and deliver a better service to their clients, elevator company Thyssenkrupp has also spotted the potential of MR when it comes to taking its business to the next level (pun intended).

Adding HoloLens to the toolkit of the company’s 20,000 field service engineers is helping them work more efficiently and safely.

For a start, they can now view detailed 3D models of elevators and escalators they are required to visit before they leave the office, identifying any areas of concern ahead of time. Engineers are also able to access historical notes for each installation and even refresh their knowledge using HoloLens to access video tutorials. During the visit, they can share the project live with in house subject matter experts for advice and guidance as they work.

The result is an engineering team who are better prepared and better supported on site than ever, with the knock on effects of slashing the amount of time it takes to conduct repairs – often from hours to a matter of minutes – and, in many cases, reducing the number of visits required too.

A catalyst for innovation

These stories are just some of the reasons MR is already having such a powerful and positive impact across both public and private sectors. Yet what’s really exciting is that many businesses are now taking the existing technology and using it as a jumping off point for their own innovation.

We have demonstrated our commitment to driving HoloLens growth through the community of developers and commercial partners around the world who are using MR to change their business. This has been done by welcoming select partners to the HoloLens Agency Readiness Partner programme and service integrators (SIs) and digital agencies from around the globe to our Mixed Reality Partner programme.

For example, software development company Black Marble has come up with a way to digitise modern policing as a direct consequence of using MR. Its tuServ application provides a whole new way of approaching the scene of a crime by letting investigators place virtual markers and gather multimedia evidence without disturbing the physical site or risking tainting evidence. Plus, they can continue to revisit potentially vital clues throughout the case in a far more interactive and stimulating way than simply staring at a crime scene photograph.

With healthcare being one of the most prominent sectors for use of MR technology, Fundamental VR’s HoloLens solution brings collaboration and medical training into a new dimension. Virtual operating theatres were once a thing of the future but we are now at point where this is becoming a reality. Aspiring doctors will be able to view the different layers of the body (skin, muscle, blood vessels etc) by wearing a HoloLens headset.

Likewise, immersive technology company REWIND is using HoloLens to help people experience one of the planet’s fastest and most exhilarating sports like never before. Due to the level of risk involved in the Red Bull Air Race, only one plane is allowed in the air at any one time. For spectators in attendance on the ground, this can make it hard to judge one racer’s performance against another’s. REWIND’s MR experience, Flight Deck, solves that problem by letting HoloLens users watch each plane race for real through their headset while holographic versions of previous competitors fly alongside as if they are going head-to- head. This is making the whole experience more exciting and informative than ever. Imagine what similar applications may one day exist for the likes of athletics, diving, Formula One, cycling and more.

Just the beginning

Mixed Reality is undoubtedly one of the most advanced and exciting technologies of this – or indeed any other – generation. The possibilities it offers for improving the way we, as a society, care for the sick, educate the young, design our cities, entertain ourselves and unlock new frontiers of success for our businesses are almost limitless.

Which, in many ways, also brings me full circle. After all, this is not about creating something completely virtual or new, or departing entirely from the real, physical world.