Skip to main content

Microsoft's new Mixed Reality Capture rig is totally wild

Hands down the biggest problem all forms of VR have right now is showing someone who isn't in the headset what is going on. Putting a 2D image on a screen that shows me what you see out of one eye doesn't do much for me. It doesn't give me the emotional feel, no sense that you as the player are actually in the world you are viewing. It's a real problem, and a lot of very smart people have found clever ways to solve this in some cases with depth-sensing cameras and green screens. The results are a little mixed, and difficult to reproduce in every game, but still very cool.

Microsoft has its own solution. Originally, this solution was a way to show everyone what was happening inside of Hololens. It has been adapted for occlusive Mixed Reality headsets, and the results are ridiculously cool. Unfortunately, it's not something any of us are ever going to be able to do on our own.

This big, complicated box of wires here has three different very important parts. At the top, you have Hololens to act as a sort of virtual camera. Wherever the Hololens is in the real world, that's where the camera will be recording from in the virtual world. Under that you have a big DSLR camera, which is capturing a high quality video of the person in the real world. In this situation, Alex Kipman.

Under that camera you have a Kinect sensor. That sensor is grabbing a depth map around Alex and editing out everything but him. This means there's no need for a green screen, because the Kinect is doing all of the heavy lifting here. All of this video and real-time editing comes together to create what appears to be real-world Alex standing inside the virtual world.

The end result is just plain cool looking. Alex is represented as a silhouette in the virtual world so you can see as much of his "house" and the things he is doing in Mixed Reality. This capture method creates presence, it helps you feel closer to how Alex is feeling in the headset. You can see him physically reach out and interact with the virtual world. If he were playing a scary game, you'd know exactly why he jumped instead of just seeing a shaky camera on his face. It's a much more complete picture.

It's also incredibly challenging from a technical perspective. Not only is this rig prohibitively expensive and incredibly complicated, it requires a great deal of precision to hold the rig steady and perfect lighting to Kinect can do its job effectively. It is unlikely we'll see a way to recreate this effect in the home anytime soon, but it is one of several tools available to developers and Holographic Academy attendees at Microsoft's Reactor space in San Francisco.

Russell is a tech nerd who chases the best of everything, from phones to game consoles to laptops and everything glowing or beeping. He's the Managing Editor of gaming content for Mobile Nations and can be found contributing to all of the Mobile Nations sites. Reach out on Twitter!

21 Comments
  • Pretty nifty setup they put together.  Thanks for sharing this.
  • Yea. I can see me getting one of these for XMas. No problem taking out a second mortgage. So where is that 3D scanner software running on a cell phone (Lumia 950XL, BTW) over two years ago? Oh yea, no Windows Phone anymore. So where are the iOS and Android versions? Everything else is on there, including Apple and Google's own AR/3D stuff. Nice MS is showing off what might be. To bad the competiton is delivering now.
  • They kept it for the surface phone.
  • Who cares. Seriously. Why should consumers pay attention to new products that Microsoft puts out? It's probably going to get cancelled soon just like every new and cool tech that this company puts out..
  • MS is more on business solution than consumers, Ford already been using MS mixed reality for quite awhile.
    Btw, hater always hate.
  • Why should anyone care ....Microsoft will soon be dead they will spin off xbox and kill any innovative products they develop in the consumer space....**** nutella bring back Ballmer ...sent from my note8
  • https://www.quora.com/What-are-the-best-decisions-ever-taken-by-Microsof...
  • It's bananas? B-A--N-A, N-A-S
  • Cool, i am able to post comments now.
  • What about the movies? 😅
  • Dslr? Pfft, that’s a RED camera. The price of about 70 dslr’s Give or take 😂😂😂
  • This! On the other hand that great deal of precision to hold it steady is actually being made easy by the Steadycam setup they are using. While some operator skill is required to operate the Steadycam it's not really that difficult but the result is amazing.
  • And how about those 5 boxes stacked horizontally off the back of the rig?  Or that red box underneath at the front?  Seems like they've gotten one of the three main components wrong, and missed at least another 6.  How am I supposed to reproduce this with such limited information? :-D
  • Those are all to enable a production environment (e.g. long ass cables, etc..). They are: a power supply box, a yellowbrik SDI to fiber converter, a Kinect for Windows power supply box, an Icron USB 2.0 to fiber converter, a USB hub to charge and get data from the HoloLens, and the red box is a Decimator cross converter for the steady cam operator's monitor.
  • Thanks! :-D
  • What the brainless sheeps Svenj, felipe214, Nolongerawindowsfan in the comments do not get is that Microsoft is often the one showing things off, complicated or not. others are looking at it and will most likely follow in their steps. If it weren't for MS, nobody would attempt it and perhaps look silly or fail or succeed and get this technology going. The world needs companies like this to keep us going forward.
  • @ISO_177.   WRONG!
  • MS is the most innovative company this year, I confidently said.
    Satya is a great CEO, he bring up MS value in just a few years.
    At least he is much better than TIM.
  • but TIM is a proud gay
  • > It's also incredibly challenging from a technical perspective. Not only is this rig prohibitively expensive and incredibly complicated Sign. You do realize this setup is open source and thoroughly documented?  https://blogs.windows.com/devices/2017/02/13/spectator-view-new-tool-hel... Even before this was documented, we reversed-engineered the setup ourselves. It really isn't that complicated.  
  • Where's Ranger Stacey?