Microsoft Research's hand gesture recognition tools pick up where Kinect left off

By John Callaham
published

The Microsoft Research division continues to work on motion gesture recognition projects similar to the company's efforts with Kinect for its Windows and Xbox platforms.

In a blog post (opens in new tab), Microsoft talks about how researchers at its UK labs are working on new tools, called Handpose, for hand gesture recognition for use with a virtual reality headset:

The system, still a research project for now, can track detailed hand motion with a virtual reality headset or without it, allowing the user to poke a soft, stuffed bunny, turn a knob or move a dial. What's more, the system lets you see what your hands are doing, fixing a common and befuddling disconnect that happens when people are interacting with virtual reality but can't see their own hands.

A similar project is being worked on at Microsoft's Advanced Technologies Lab in Israel. It's goal is to replace PC mouse movements with hand and finger gestures. The team recorded millions of hand images and then used that data to come up with every single possible hand pose or motion:

The goal of the research project, called Project Prague, would be to provide developers with basic hand gestures, such as the one that switches a computer off. And it also makes it easy for developers to create customized gestures for their own apps or other products, with very little additional programming or expertise. The system, which utilizes machine learning to train systems to recognize motions, runs using a retail 3D camera.

The team hopes that the tools created for Project Prague could be used by apps so users can, say. flip through pages of a PowerPoint presentation by using their hands movements.

John Callaham
John Callaham
27 Comments
  • So Kinect is not dead just being used in other ways.
  • Kinect is dead this is their new toy
  • Kung-Fu for Kinect just came out for Xbox One three days ago and it's awesome.
  • @coip, I'm glad there's a new Kinect game you like -- I know you've been pretty unhappy with Microsoft's handling of the Kinect. Do you think they've turned the corner on that, or at least or no longer accelerating in the wrong direction?
  • The new Kinect game is from an independent developer, Virtual Air Guitar Company, and has nothing to do with Microsoft. I'm more disappointed than ever at Microsoft for their handling of Kinect--a device the Xbox team hasn't mentioned at a major conference since 2014, a device Microsoft Studios hasn't released a game for since 2014 even though executive Shannon Loftis promised in 2015 that first-party Kinect games were on the way and now claims she's "not sure if they're on the way", despite the fact that she's the head of Microsoft Studios. They only thing Microsoft has done since 2014 in terms of Kinect is forcibly remove features that some of us made purchasing decisions based on (i.e. Kinect gesture, infrared controller assignment, etc.), and have removed the Kinect port from the upcoming Xbox One S. Not very admirable from a company that launched Xbox One by declaring that "Kinect is Xbox One" and that it would never be decoupled. I just don't like being lied to and then, when things change, get strung along and they equivocate and demur instead of apologizing and being earnest. It's not fair to their customers. Props to Virtual Air Guitar Company for continuing to support Kinect when Microsoft won't, though. And the good news is that they stated they will continue to do so, with another Kinect game coming this December. It's nice to see independent developers continue to innovate. It's a shame, though, that their newest game, Kung-Fu for Kinect, isn't getting any press attention (including from Windows Central), as it's really quite well done for an indie game.
  • That's too bad. I also bought the original XBox with the Kinect. In my case, I bought it for the games and voice control more than the gestures (and because back then, you couldn't get XBox One w/o Kinect). As a game system, I couldn't be happier with it -- Fallout 4 is my main game right now and will probably hold that position for another several months. I don't personally miss the lack of features, because I'm not sure I would have used them. I both liked and resented using gestures on the menus -- thought it was incredibly cool to be able to do it on those rare occasions where I needed to do something and didn't have the controller handy, but also found that it would just pop a hand up on the screen sometimes when we were sitting watching TV and it didn't appear anyone had actually raised their hand. It does seem that the same update that removed gesture controls also made the voice recognition much worse. I can't really turn my TV on or off by voice command anymore and it used to work almost flawlessly. In any case, I agree with you that Microsoft should not have gone back on their promise. That's just poor customer service and throws their future promises into question. I cut them a little bit of slack because of the timing -- concurrent with the Balmer-Nadella transition and attendant management shuffling, so there's inevitably going to be some lack of continuity, but as you point out, that doesn't really excuse breaking company promises that drove customer purchase decisions. The new CEO should still try to keep those where possible and tell his management team to do the same. @WindowsCentral, could you cover games like Kung-Fu for the original XBox One customers who did buy it with Kinect?
  • I guess we can now add this to the list of Microsoft alienating Kinect fans, as just an hour ago they've announced they're forcibly shutting down Xbox Fitness and removing customers' access to the in-game content they've already purchased. Unbelievable.
  • I'm so happy someone out there feel's the same way i do about all of that, my friend, i must add you to Xbox if at all possible, my GT is CJ Fawx, if you wish, loved every bit of that!!
  • I think that they are going to relaunch kinect as something else. It will be a device that is compatible with both pc and Xbox. One device for all.
  • Kinect was never dead, the problem is that the industry never took advantage of the new technology. They rather be lazy and reiterate the kind of tired gameplay that's proven to sell well. I'm afraid this will still be the problem with current-gen VR. There needs to be a change in the minds of developers.
  • Kinect + VR gaming = made for each other. The next XBOX would hopefully get people, and developers' to actually use the Kinect this time. Kinect would allow people to not have any controllers for VR gaming. Possibly just a single handheld remote with some buttons to toggle weapons/stuff.
  • YES! I had this thought awhile ago too but I never imagined VR gaming would take off so quickly back then.
  • I'd be broader about it and say that any hand recognition technology is a good pairing with VR. Not only Kinect but there's also the Leap controller, and Intel's RealSense cameras (which you can see used in the video), to list only the best known ones. Eventually they will converge.
  • there is another video on Microsoft Research's YouTube Page that shows a "Simulated" Holographic Room That can Project Interactive 2D Content Room and User can Interact and see it without any additional Equipment(Naked Eye).
  • Link?
  • here you go..... https://www.youtube.com/watch?v=GYkRRbP7m8s
  • Link pls
  • Personally would just be happy to see a consumer Kinnect for PC that doesn't need a kludgey adapter and brings simple non-touch gestures and voice commands to Win10... not to mention a great low-light Skype expierence!
  • I wonder what happened to the gestures beta for wp? Did they stop the project??
  • I miss my Kinect gestures.
  • Indeed, I've actually been considering selling my Xbox One and upgrading to an Xbox 360 instead just to have gesture navigation and better Kinect game support. Had I known in 2013 that Microsoft was going to renege on every promise they made with Kinect on Xbox One, I never would've gotten an Xbox One and would've bought an Xbox 360 instead.
  • Sounds like they're working on a lot of stuff at the moment, I cant get used to the vr headset those fixed hands really mess with my brain.
  • It's not fixed hands here. It's your hands that can move and disappear from view.
  • Amazing to see developments in this field. I hope eventually having a RealSense camera yields results for actual use beyond facial recognition.
  • Amazing!
  • nice but i am dont chaceing the ideas. i think i need to wait at least 2 to 4 years to see if something is worth my time.
  • Just wish they would bring back the gesture controls for Xbox.