Skip to main content

Microsoft Research works on interpreting sign language with Kinect

Microsoft is taking its Kinect technology to a whole new level. We've already looked at how the company is attempting to integrate its product into retail, but Microsoft Research is also tapping into sign language. Numerous videos have been released showing off how the Kinect Sign Language Translator is able to interpret sign language and translate it.

The technology works both ways (wouldn't be such a good conversation otherwise, right?), but this alone shows just how useful the Xbox Kinect is when not used to keep fit or play video games.

How does the technology work? Sign language is detected, captured and translated to text while speech is rendered as visual signs for the hard hearing to see on-screen. Being mastered in Asia, the technology currently supports 300 Chinese sign language words out of a total of 4,000.

Check out the video below to see an overview on the project and how the technology works:

Explaining the scope of the project, Guobin Wu, Research Program Manager, Microsoft Research Asia, wrote the following in a blog post on MSDN:

"The next milestone was to build a sign language recognition system. The team has published many papers that explain the technical details, but what I want to stress here is the collaborative nature of the project. Every month, we had a team meeting to review the progress and plan our next steps. Experts from a host of disciplines—language modeling, translation, computer vision, speech recognition, 3D modeling, and special education—contributed to the system design."

​This technology can prove useful in many situations and Wu plans to make the technology available outside the lab. According to the program manager, there are more than 20 million people in China who are hard of hearing, who join an estimated 360 million around the world. We look forward to seeing further progress made with the project.

Head on over to the Microsoft Research website (opens in new tab) for more details and be sure to check out the blog posts over on MSDN (part 1 (opens in new tab)) (part 2 (opens in new tab)). via: The Next Web

Rich Edmonds
Senior Editor, PC Build

Rich Edmonds is Senior Editor of PC hardware at Windows Central, covering everything related to PC components and NAS. He's been involved in technology for more than a decade and knows a thing or two about the magic inside a PC chassis. You can follow him over on Twitter at @RichEdmonds.

  • Awesome
  • Like the Red Sox! Sorry, couldn't resist. But this is really great. How long before they can add ASL and then deaf people can sign cross-culturally -- this seems to be an ideal venue for real-time translation since the output (text) won't interfere with the input (gestures).
  • So will it recognize when you're watching porn, and when you're doing it? I mean hey, they are a sign language of their own merit.
  • Impressive, and dare I say it.. A potential life changing use of tech. Kudos!
  • Like a keyboard! they could write the text there :)
  • I wondered how long before tech would finally start catching up with signed languages. Very awesome.
  • Wait, what?
    That's cool and all, but how about more language support for speech recognition... :/
  • Different research and development group. Don't worry, they're still working on it too.
  • sign language is a language its int its name, also i would think sign language would be one of the more useful ones for it can really help people communicate and learn, also help in many ways   
  • Oh god lets halt something very useful for people that if I'm correct never get to communicate with others online without a keyboard so you can have more speech recognition languages..
  • This might soon become a great advantage of Microsoft and XboxOne.
  • Look one step further. Incorporate this technology into phones and tablets. The ability to perform this type of translation on the fly is its true potential.
  • This is very cool. MS do come up with some cool stuff, and I wonder how long it will take for competitors to catch up with all the technical debt in all the different areas MS are exploring.
  • I am going to spend the next few hours emailing as many deaf associations in as many countries as i can with a link - spread the word (or sign) this is worth more than all the games in the world!
  • Good on you.
  • I patiently wait for the day an article comes through titles Microsoft research looking to implement Kinect gestures into windows phone. If Sammy can do it why can't the creative minds that brought us the Kinect do it.
  • I think Microsoft need to shrink the size of kinect. Make it one of the feature on their wearable tech like Microsoft glass(es). I can see this can be very helpful.
  • Finally a useful way to use Kinnect other than for gaming. I hope Microsoft makes this mainstream, it would help millions of people.  
  • You mean the ones that cant use a keyboard instead?
  • No... Like the ones who prefer multiple input methods like speech and touch in addition to keyboard and mouse.
  • People have already done some amazing things with the original Kinect hardware. There is a guy out there who is a DJ and he controls his laser props during shows with a Kinect.
  • I remember years ago when Microsoft purchased the Kinect Technology with the chips inside the original device it could ready sign language. Everything the Kinect does now has been replaced with software.
  • Just my two cents, but those of you who keep referring to keyboards to me seem to completely miss the point of this advancement, a key board does not allow true interactive communication, it requires that both parties stare at a monitor rather than look at each other, communication is so much more than words, its facial expressions, body language, posture ECT, why do you think video chatting and video conferencing is so useful, its because it adds so much more to the conversation, it allows you to understand context, meaning, express emotion, things that written (typed) words just don't allow, typing restricts emotional context, this technology is truly amazing and I tip my hat to the people and organizations that are making it happen
  • Someone who gets it. I know a fair number of people that sign who would love this technology.
  • Well said sir!
  • Is that Steve Ballmer in the background?
  • Awesome ill deff buy one to support the movement
  • This is innovation! I've loved following Microsoft Research for a long time and its just neat to see some of it come to fruition.
  • If they can extend this to include all sign languages, then great.
  • I think that will be a goal too.
  • Brilliant stuff, and proudly Microsoft
  • Very cool!!
  • Bad Ass!! Thank you Microsoft.
  • This might be very ignorant, but they can see, so they can type or write what they want to say instead of sign it, right?  Seems like text to speech and speech to text is much more important than sign language.