Microsoft's 2018 Mixed Reality plans show progress toward immersive digital future

Last year Kipman described how our descendants would look at the first century of computing as a peculiar time in history where humans were forced to live in 2D space. He said this brief 100-year span would be a single period in the vastness of time where we humans were entertained, communicated and managed our lives from behind a screen.

Kipman is a futurist. He envisions a connected world where the power of augmented and virtual reality eradicates distance as a barrier to being in one another's presence. He conceives of a world where holographic representations of individuals can be "teleported" anywhere, anytime, to communicate, to explore, to feel holographic environments via granular haptic feedback, and not only see and hear them.

Kipman sees a reality where ones physical self can be in one place and their digital self another. Executive Vice President, Microsoft Artificial Intelligence and Research Group Harry Shum, also a futurist, predicts a future where everyone will have an AI alter ego that represents them in the digital space. Microsoft is developing the technology to make this mixed reality and AI-driven world a reality. In fact, Kipman recently identified the following industry-defining trends for 2018 (opens in new tab):

  • MR and AI
  • AR and VR
  • Immersive communications.

Mixed Reality and AI

Surgeons collaborate using HoloLens

It's important to note the expectation-setting trends Kipman identified are tangible steps Microsoft has invested in toward the future Kipman envisions. Microsoft CEO Satya Nadella sees MR and AI as the next evolution of computing and is fully invested in their development. Additionally, Windows Mixed Reality as part of Microsoft's device-agnostic Windows 10 OS, is foundational to the company's platform approach to democratizing MR.

Kipman stressed MRs ability to help people transcend space and to enable interaction with digital counterparts. Surgeons from various locations demoed this capability last year as they donned HoloLens' and collaborated during a surgery (above video).

That was admirable but Kipman's vision for intelligent MR required advancements in HoloLens' onboard AI. HoloLens must be capable of comprehensively perceiving and understanding our actions and intent, our environment, our interactions with it and how the digital world it manifests should respond to us. Toward that end last year Microsoft announced it was adding a dedicated AI coprocessor to HoloLens' custom Holographic Processing Unit (HPU).

This AI coprocessor enables native Dynamic Neural Networks (DNN) which approximate how human brains work. This is important to the advancement of computer vision and other recognition tasks required for HoloLens to "understand" its environment. Marc Pollefeys (opens in new tab) director for science at HoloLens adds:

This is the thinking you need if you're going to develop mixed reality devices that are themselves intelligent. Mixed reality and artificial intelligence represent the future of computing, and we're excited to be advancing this frontier.

Augmented and virtual reality

Microsoft's AR efforts have focused on collaborations with the enterprise, private and education sectors toward developing custom applications for HoloLens. Various industries like healthcare, entertainment, NASA and more have partnered with Microsoft to bring AR to its customers. Furthermore, HoloLens' AR tech, like VR is meant to be emulated and democratized by OEM partners.

Microsoft's VR efforts are more consumer-focused as it continues expanding its VR platform as partners bring VR headsets to market. VR is still a struggling category but Microsoft and Kipman hope the masses can be won to what it is expected to be an AR and VR computing future.

An immersive experience

Immersion is critical to Kipman's vision of screen-less AR and VR computing. He believes a "killer app" for MR will be a social communication app that tears down the walls of isolation. Current communication methods provide 2D "screened" experiences at best. Kipman envisions a MR platform that replaces 2D communication with immersive "virtual presence" sharable experiences.

Kipman shared how in work or life, communication and presence, are the fabric that ties us together. He shared how he wants his daughter, in the U.S., to be able to "physically" play a board game with her cousins in Brazil. That technology is in its early stages today.

Kipman was challenged regarding the HoloLens' limited field of view (FOV) to which responded, "ultimately that's the wrong question." He said,

That's the equivalence of showing someone holograms for the first time, and you then saying, What's the size of your television? The field of view is almost irrelevant, what we should be talking about is the density of light or radiance that shows up.

Still, the "picture frame" FOV is a current technological limitation preventing the fully immersive AR experience Kipman describes. Of course, technology will evolve, and one day AR holograms may engulf us as entirely as VR does in many scenarios.

Collaboration, AR glasses and AltSpaceVR

Kipman's vision for collaboration requires a foundation that connects people in life and business. Microsoft Teams was demonstrated capable of connecting users across different devices and experiences, including AR via HoloLens as users worked on a car (above video). At the other end of the spectrum is Microsoft's investment in AltSpaceVR, a social platform for VR. Both of these tools are being evolved to provide the foundation for robust collaborative AR and VR experiences.

As we move forward, Kipmans vision of smartglasses that transition from opaque to clear to accommodate AR or VR scenarios may become the norm. Furthermore, as more devices become part of the cellular roadmap, and the power of Windows PCs makes it to increasingly non-traditional form factors, perhaps in time Kipmans and Windows Insider Chiefs, Dona Sarkar's, vision of smartglasses as smartphone replacements will become a reality.

Smartglasses as smartphones - your social and privacy concerns won't matter

If in the future smartglasses become the norm, humans will exist in that "screenless" world Kipman envisions where we are surrounded by and can interact with our data and where our digital selves are persistently active on a digital plane. It may sound like fantasy, but the trends Kipman suggest we watch in 2018 and that Microsoft is investing in, are moving in that direction.

If you want to jump on the MR bandwagon now's the time. We'll have to wait and see whether Microsoft can achieve its aims long term.

Save $100 on Samsung's HMD Odyssey

Jason L Ward is a columnist at Windows Central. He provides unique big picture analysis of the complex world of Microsoft. Jason takes the small clues and gives you an insightful big picture perspective through storytelling that you won't find *anywhere* else. Seriously, this dude thinks outside the box. Follow him on Twitter at @JLTechWord. He's doing the "write" thing!

  • I personally will consider owning and using a hololens type device, but I won't even think of using any WMR headset. Anything that isolates me from my surroundings is impractical for me.
  • I think when the technology advances enough, it'll be a one device for all purposes (AR and VR) scenario. I do think there are specific and separate use cases for both VR and AR, though. I'm excited for the tech to advance.
  • It is, so I understand your point. That said, if you never tried a WMR headset, give it a spin. It's an incredible experience.
  • Depressing to say, but the thing that will drive any of that quickly will be pornography.  The whole "virtual socialization" isn't that big a driver in and of itself.  But it should be pretty obvious that telepresence involving physical feedback is going to be all about the porn industry.  That mess aside, I see AR being more useful in general situations because it brings virtual items without disconnecting us from the physical reality.  My wife gave me an Acer WMR rig for Christmas, thinking it would be something I'd really make use of.  And, to be honest, I do--but for one thing only, playing Elite Dangerous.  And even at that, WMR is an incredible hassle to make work well and consistently.  I dread every time I have to spin up that idiotic cliff house.  Talk about absolutely NON-productive.  Nothing you do in there is anywhere remotely as efficient, useful or even visually appealing compared to working with a really nice screen (or screens), a keyboard and a mouse.  Nothing.  And having to "walk" around the cliff house just to throw things up to work with them is insane.  Care to guess who has a better idea, if they'd only let users STAY in the environment?  SteamVR.  And I despise Steam.  But in the few minutes you're waiting for THEIR idiotic cliff house to spin up you're sitting there in white round room.  A clean, perfect canvas for throwing windows up.  Oh, sure having a soothing 'paint' on the wall would be fine, but that's a much more efficient way to work with apps than some house.  Apps aren't paintings on a wall.  But you couldn't tell that from the way Microsoft's minions seem to think.
  • From what I've gathered, there's actually a lot of projects regarding VR interactive porn. Thats besides just 360 video-like experiences. Let's face it, regarding privacy of use, VR is a great space to watch anything you want to keep from others.
  • While other VR and AR providers are focused on Gaming and entertainment for the most part, it's good to see MSFT leading the way with WMR to be used in all the aspects of our lives.
  • Exactly, because this is what will take it beyond just being a gaming gimmick. Kipman has a vision.
  • For the love of science and physicists everywhere, please, PLEASE stop using "holograms" and "holographic" when referring to 3d models presented in virtual and augmented reality. There is nothing "holographic" about the way Microsoft's technology works - they're simply projecting a 3D model on a 2D piece of glass. A proper hologram is a true, volumetric image in 3D space which is capable of being viewed without the use of special lenses or other optics. Microsoft's marketing department would like you to believe otherwise, but we are nowhere near proper 3D "Star Trek/Wars"-style holograms yet. 
  • Or X-Men's Danger room. Hey, I get it.😉
  • Physicists? Why them? 3D models on a 2D piece of glass? What are you taking about? If you're expecting science fiction level technologies to appear right before you today, perhaps you should consider putting yourself in a Cryosleep chamber for another 20-25 years.
  • My pet peeve is with all these new technologies that will somehow enable remote surgery. Usually found in every article about 5G, but made its way into this one today. There is a 0.0% chance of that becoming an actual thing. No one but no one is going to allow a surgeon to operate on them from 1000 miles away. If by chance you are insane enough to think you'd be the a willing patient for that procedure, would you let your children? Such ridiculous.
  • It's a metaphor, not unlike calling certain UI elements "buttons" or "sliders", not to speak of "windows".