Microsoft Research gets Kinect to notice your fingers with 'Handpose' project

The Kinect is at once a niche and – forgive the pun – game-changing technologies. Although it has not quite resonated yet with customers in actual gaming, the potential is there for some fantastic experiences in the coming years.

One issue with all systems that detect motion is accuracy. Version 2 of the Kinect is certainly more powerful than its predecessor, but the nuances of finger movement are still lacking. Enter 'Handpose' a project that Microsoft Research has been working on for some time.

The researchers combined 3D hand modeling with machine learning to "to teach the computer how to infer hand poses".

So what are some possible uses for such a technology? In a detailed blog post, Microsoft suggests these scenarios:

  • Artificial Intelligence - Helping computers interpret our body language, including mood and pointing at objects
  • Video games – Reaching out and grabbing something instead of using a controller
  • Sign language translation
  • General computer tasks – Email sorting, manipulation of objects on screen aka the 'Minority Report' experience

Interestingly, the article does not mention Microsoft's HoloLens, which relies heavily on hand gestures for clicking and object manipulation. It would seem from our experience that HoloLens at least makes some use of this technology, and if not, will do so in the future.

Watch the above videos that explain how it all works or read the detailed article from Next at Microsoft.

Source: Next at Microsoft (opens in new tab)

Daniel Rubino

Daniel Rubino is the Editor-in-chief of Windows Central, head reviewer, podcast co-host, and analyst. He has been here covering Microsoft since 2007 when this site was called WMExperts (and later Windows Phone Central). His interests include Windows, Microsoft Surface, laptops, next-gen computing, and for some reason, watches. Before all this tech stuff, he worked on a Ph.D. in linguistics and ran the projectors at movie theaters, which has done absolutely nothing for his career.

  • Kinect..Right on top of the list of microsoft features implemented and forgotton and ignored
  • Maybe by you, speak for yourself...
  • Sorry.. I am not talking about consumers here I am talking about Developers. You couldnt even get proper bundles with kinect anywhere.. After the backfiring with sales and the cheaper PS4 it all went down for kinect.
  • You are correct in that Microsoft really dropped the ball in terms of promoting Kinect and keeping it in stock since the unbundling last June and the launch of the standalone Kinect last Fall. They didn't even mention the word Kinect at E3 or Gamescom. They did absolutely zero promotion for some fantastic Kinect games--D4, Dance Central, Fantasia, Fruit Ninja Kinect 2, etc.--and they've included it in almost none of their X1 bundles since August, and it's been mostly out of stock at every major retailer for months.
  • Agreed. I think Phil Spencer is doing a terrible job getting a wide audience excited about Xbox. He seems to just be acting as a subordinate of the Windows team and isn't really doing anything to distinguish Xbox One from the crowd. Spencer seems satisfied to play second or third fiddle to Playstation and Steam. Would love to see someone much more ambitious in charge of Xbox, someone interested in moving the platform far ahead of the competition as they were when they developed Xbox Live and Kinect. The way he downplays every new technology makes him seem like the short sighted wet blanket of the game industry.
  • I use it everyday.
  • I wish I had... But getting my box without one just meant I could get it sooner...
  • Oooo what type of box? XD (sorry. It needed to be said XD)
  • Why the dumb comments usually come to the first?
  • Kinect is going to have a lot of competition from Intel's RealSense technology. Intel can fit their system into a cellphone, Kinect still requires that an extremely large device be attached to something and wired to a computer. What I would like to see from Kinect 3.0 is full 360 degree tracking, so that once I "log in" to the Kinect it can follow me around the room
  • Kinect is way, way, way more advanced and powerful than Intel's RealSense.
  • leap motion? :P
  • Uses? Naruto hand signs in game :D
  • Thumbs up while watching a show with the facebook integration
  • LOL! that'd be extremly hard to do as they do them so fast in the anime...that still would be cool to use jutsus in a game like ninja storm
  • Hahaha I know, but would be cool to be a copy ninja for a while ;)
  • Haha! Nice.
  • Oh yeah would be awesome if they worked that out...! :D
  • Air Gestures for the next Lumia flagship?
  • That picture doesn't show much progress.
  • What do you mean!? Both hands have fingers...
  • The guy has his fingers together, on the screen they're spread out...
  • Details, details, details...
  • Needed to be pointed out... anyway, who cares? right?
  • Ikr xD
  • Ha you beat me to it!
  • MS needed to develop a killer kinect game around the time of the XB1 launch to really showcase the technology and get consumers excited about buying the more expensive console, but unfortunatley, they gave us nothing to do with our expensive add-on.  Now the only thing i use mine for is voice controls, which could have been handled by a built-in microphone on the controller, like the Fire TV.  I still think they should implement something like that in the newer consoles, a small mic in the control so that everyone can benefit from the voice controls, which i find work fairly well. 
  • Plenty of great Kinect games have launched on Xbox One: Xbox Fitness, Kinect Sports Rivals, Dane Central Spotlight, D4, Just Dance, Shape Up, Fantasia, Blue Estate, Boom Ball Kinect, Fruit Ninja Kinect 2, etc.
  • Yeah while I agree Microsoft should have done more than just Kinect Sports Rivals (that was originally a launch game), there are a decent amount of good Kinect games out there now. It is disgraceful though that Microsoft has only published 1 Kinect game since the launch of the console. The support for Kinect and now the disasterous support of Halo the Master Chief Collection (both campaign and multiplayer are still bugged out) makes me question the leadership and competency of the people running Xbox.
  • I feel the problem with kenect is that it hasn't proven to be what people pers eve it to be. Meaning they want it to actually allow them to be in the game with all there movements and with games people play like cod and battlefield. I think it may get there but not before it gets past the people who say that's to realistic. This is my thoughts as well as what people I know have said.
  • Just in your imagination.
    Keep dreaming dude...
  • Is everybody left-handed at Microsoft Research? 
  • Microsoft should create an Xbox app similar to VoiceStudio except focusing on the Kinect camera to learn new motions and enhance its prediction and reliability.
  • The issue that I have with Kinect if it's fixed, would be major leap.  During navigation whether thru swipe gesture or pointing aka clicking, Kinect need to narrow the gesture movement.  For example, swiping gesture felt like moving my right arm across toward my left arm can get tiring sometimes.
  • Try recognizing my fingerprints and I'll be impressed.
  • Total fail I, the picture. The guy's fingers look nothing like what Kinect identified...
  • Waiting on Cortana and windows 10 for my xbox one and Kinect.