Skip to main content

How Microsoft and Volkswagen partnered to get HoloLens 2 to work on moving vehicles

Microsoft Hololens Vw Warning
Microsoft Hololens Vw Warning (Image credit: Microsoft)

What you need to know

  • Microsoft worked with Volkswagen to get the HoloLens 2 to work in a moving vehicle.
  • Usually, mixed reality headsets get confused when in moving vehicles due to mixed signals from sensors.
  • Microsoft and Volkswagen started collaborating in 2018 to create the moving platform feature for HoloLens 2.

Microsoft partnered with Volkswagen to get the HoloLens 2 headset to work while in a moving vehicle. Up to this point, mixed reality headsets have been unable to work when on the move due to conflicting sensor data. Now, the HoloLens headset can work while on large ships, and Microsoft is working on expanding to more vehicles in the future.

The main challenge that prevents headsets like the HoloLens from working when on the move is that sensory input can cause confusion. The visible-light cameras and the inertial measurement unit (IMU) gather data for the HoloLens. When information from these sensors conflicts, the system doesn't work properly. Microsoft equates this to humans getting seasick.

Microsoft addressed this issue by creating an algorithm that models any differences between a headset's sensors.

Microsoft Hololens Vw Navigation

Source: Microsoft (Image credit: Source: Microsoft)

Volkswagen and Microsoft started working together in 2018 to enable the HoloLens 2 to work within a moving vehicle but ran into logistical challenges along the way.

The global pandemic presented a unique roadblock that made it difficult to test prototypes. "We had to do a lot of testing in my apartment. These aren't ideal development conditions," said Joshua Elsdon, a Microsoft senior software engineer. "All of this stuff was done remotely and distributed across different countries, which was interesting."

Eventually, more testing was done near Microsofts' Redmond campus. Initial tests in the area were done on Puget Sound, which is a body of water near Redmond. Later, Microsoft worked with Volkswagen to test things out within vehicles.

"We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car," said Michael Wittkämper, augmented reality expert at Volkswagen. "This [opened] up completely new possibilities to not only display holograms within the driver's forward-facing field of view, but also wherever the user wearing the glasses is looking."

This isn't Volkswagen's first foray into mixed reality. The automaker introduced a head-up display in 2020 that projects navigation arrows, lane markings, and other information onto the windshield of a car.

At the moment, the moving platform feature for HoloLens works with large ships. In the future, Microsoft plans to get it to work with elevators, trains, cars, and more.

Sean Endicott
Sean Endicott

Sean Endicott is the news writer for Windows Central. If it runs Windows, is made by Microsoft, or has anything to do with either, he's on it. Sean's been with Windows Central since 2017 and is also our resident app expert. If you have a news tip or an app to review, hit him up at sean.endicott@futurenet.com.

7 Comments
  • Microsoft's next acquisition
  • Interesting news considering the commotion about HoloLens 3, the meta verse, etc. Would really be a shame if this project got canceled.
  • Article is not clear why VW is needed as opposed to a car or bus or anything moving really.
    3D Maps look horrible, Google can implement that at a drop of a hat (no pun).
  • I don't think VW was needed over another automaker that would have done the same thing. VW helped test and develop the feature, as noted in the article.
  • So weird what goes on inside big organisations like Microsoft. In March the news was that hololens is dead, military contracts are falling apart, teams are is disarray and it's all basically snafu... By May, they're signing up new clients, launching new experiences and revamping app stores. So what gives? Is it dead, alive or zombie?
  • It's very much alive, albeit in a very hamstrung fashion due to 1)bean counting and 2)petty office politics. Unlike bureaucracy in govt departments things can move alot faster in a company providing there is a will to do so (as well an impetus to do so). So, in the case of hololens, if Microsoft doesn't provide more resources to the hololens team as well as to the team working on UWA, ARM64EC. They are going to lose a massively lucrative government contract. Once one is lost due to failure of delivery (especially due to bean counting), they can say goodbye to future government contracts. As Amazon, Google and Apple will beat Microsoft constantly with that failure every chance they get leaving Microsoft without any form of credibility. Thus Microsoft will forever have to wear that shame. The bean counters got them into this mess - in order to inflate the share prices they bidded aggressively for the DOD contract. So, unfortunately due to them Microsoft cannot back out now.
  • The guy is working remotely from his apartment, article says so.
    When he gets to his desk he'll realize it's dead and has to find work.