Why Microsoft must bring sign language recognition to Windows and Cortana

Microsoft's inclusive design mission is guiding the company in ensuring its products and services are designed from conception forward with every user of all ability levels in mind. Though the company has made admirable progress in this regard, it is still a work in progress.

Microsoft's Seeing AI app helps people with blindness, Project Fizzyo supports children with Cystic Fibrosis, the Emma Watch and Project Emma aids people with Parkinson's Disease and Microsoft's Immersive Reader helps children with Dyslexia. There are millions of people with varying levels of abilities who are either excluded from interacting with the technologies of modern society or whose physical limitations prevent their full participation in everyday tasks.

Microsoft has embraced the challenge of creating specific solutions, like the tremor-halting Emma Watch, which targets a particular aspect of a disability. It has also incorporated solutions that level the playing field into its technologies, like gaze control in Windows, which enables people with immobility to navigate the OS. Given this integrated solution for people with para- or quadriplegia, a similar OS level solution that enables Windows or Cortana to understand sign language for the 466 million people with disabling hearing loss, in a world where "speaking to AI is becoming the norm" seems like a natural goal for Microsoft. And given that a developer "modified Alexa" to do just that we know that it's also possible.

If Alexa can do it so can Cortana/Windows

Developer Abhishek Singh created a web application that uses a camera to view and understand sign language which is then translated and spoken and heard by Alexa via Amazon's Echo. A typed response is then provided by the system that can be read by the user after Alexa speaks her response.

Using machine learning platform Tensorflow, Singh trained an A.I. to understand American Sign Language and used Google's text-to-speech to translate the signs into spoken words. Singh said, "The project was…inspired by observing a trend among companies of pushing voice-based assistants as a way to create instant, seamless interactions."

Given Microsoft's A.I. and machine learning investments, its Cognitive Services that recognize human faces, activities, speech and more and the role of the camera in Windows PCs for biometrics Microsoft has the end-to-end resources to create a system that can communicate with users who use sign language.

Inclusion is what Microsoft is about

Most companies have some degree of dedication to inclusive design. Microsoft is not unique in that regard.

Microsoft is unique, however, in that its CEO Satya Nadella is personally driven toward inclusion goals due to his experience raising two children with disabilities including his son Zane who has severe Cerebral Palsy. This has led Nadella to promote a pervasive empathy mission throughout Microsoft that imbues its inclusive design efforts with a depth of sincerity, a level of detail and broad scope that makes it stand out in the industry.

Windows now has gaze technology as a result of a hackathon that enabled a former NFL great with Amyotrophic Lateral Sclerosis (ALS) play with his son despite using a wheelchair. Immersive Reader is integrated throughout Microsoft's cross-platform products like OneNote enabling school systems and parents to support children with Dyslexia and other disabilities. Sign language recognition built into Windows/Cortana would be a systemic component of the platform that fits naturally with Microsoft's other efforts.

Including sign language recognition is a must

Cortana navigates a meeting with a deaf person in attendance.

If Microsoft's products are meant to be used by everyone then sign language recognition must be a part of the equation. Though the companies digital assistant and smart speaker efforts have paled in comparison to the success rivals are enjoying Microsoft still has a case for making this move.

At build 2018 Microsoft demonstrated Cortana's navigating a meeting, transcribing the conversation, responding to participants and providing text that a deaf/hard of hearing participant could read. A natural progression to such a scenario would be including sign language recognition for people with deafness who do not speak or others who rely on sign language who are unable to speak.

Microsoft's partnership with Amazon by bringing Cortana and Alexa together shows that it is serious about keeping its ambient computing efforts visible in the consumer space. Perhaps bringing sign language recognition to Windows and Cortana could be boosted by similar or joint efforts integrated into Alexa. However it pans out, it just needs to happen. I hope Microsoft agrees.

Jason Ward

Jason L Ward is a columnist at Windows Central. He provides unique big picture analysis of the complex world of Microsoft. Jason takes the small clues and gives you an insightful big picture perspective through storytelling that you won't find *anywhere* else. Seriously, this dude thinks outside the box. Follow him on Twitter at @JLTechWord. He's doing the "write" thing!

  • Frankly I would be happy if they just made tablet mode better or fixed problems with updating. Aka, Microsoft has a LOT of things it MUST do to improve Windows, sign language req is pretty low on the list.
  • Maybe for you, but there is a large segment of the population that sign language recognition would help.
  • Hi Rann: Thanks for your response. As your attention to Window Central articles will show, we acknowledge MANY things Microsoft can do to improve Windows, improving tablet mode being just one of those things🙂. "Improving Windows", however, is not the general theme of this article, however. I think you may have missed that. 🙂 Hopefully you'll recognize that the theme of this article is more that Microsoft MUST bring sign language recognition to Windows and Cortana to better serve the deaf/hard of hearing/voice less community as part of Microsoft's inclusive design mission not that Microsoft MUST do this to improve Windows. You say: Aka, Microsoft has a LOT of things it MUST do to improve Windows, sign language req is pretty low on the list. Your hierachal assessment of where bringing sign language to Windows falls in relation to a list of "Windows-improvement" features is actually not related to this article. This piece's focus is about serving an as yet unaddressed portion of the population via inclusive design, not about improving Windows. Though the service of that particular population through the suggested means would have the derivative affect of improving Windows. That improvment, however, is a secondary effect of addressing the article's focus. 🙂 Hope that helps.
  • Cortana can't understand some languages or people let alone sign language. I guess now Microsoft is looking for that niche market for Cortana. best of luck with that.
  • Interesting that apparently now there are resources available for sign language, yet struggling with supporting more regional and more spoken languages in general for Cortana
  • Cortana barely speaks English... But good luck, MSFT.
  • I'm Deaf... I think Microsoft needs to work on improving accessibility by enabling live or smart closed captioning because I would like to see Windows able to translating any voices or sounds to a closed captioning screen when I surfing websites or apps even games with players use voices.
  • I appreciate your idea Jason. A sign language Cortana indeed fits perfectly with Microsoft's STATED vision of improving people's lives. The only problem is, I think Microsoft is full of crap. They talk a mean game, sure, but I think at heart they are just the same greedy corporate monster they've always been. Let's face it, if they were really concerned about more than sheer profits, they'd have a Surface phone on store shelves, niche phone or not. They'd have one if ONLY so they could realize their vision of making the world a better place without relying on the iOS and Android platforms to realize their vision unhindered. Sorry, but I just can't help but believe this touchy feely new image of Nadella's is all just smoke and mirrors. A lot of greedy corps do it. MS caring about the disabled is no less marketing dung than the oil company that "cares deeply" about the environment. Granted Nadella has a disabled son, and that SHOULD make him sympathetic. It SHOULD make him consider an idea like yours. It SHOULD make him a champion of the deaf community, sure. But unfortunately guys at his level are great at compartmentalizing things. It allows them to fire thousands of people and still wake up and like themselves in the morning. It allows them to put profits ahead of EVERYTHING else. I would happily eat my words should Nadella put his weight behind a Sign-Language Cortana. I just can't see it happening.
  • PS. That said, I still love your article. This is definitely what Microsoft SHOULD do!
  • “It allows them to fire thousands of people and still wake up and like themselves in the morning. It allows them to put profits ahead of EVERYTHING else.” Nadella is a CEO. He is SUPPOSED to put profits ahead of everything else. That is his job. If you want a touchy-feely-feel-good-because-wow-we-are-just-SO-inclusive person, then you need to hire a CTFFGBWWAJSIO. The CEO is all about business. Like it or not, Nadella is doing his job. Profits are soaring.
  • We know. It's not a radical idea to expect businesses to stay true to their word or call them out when they don't. We all obviously like Microsoft in one way or another and hope we can have a company that's also as compassionate as it markets itself.
    A profit only world is why ethical consumption under capitalism is next to impossible. Also, no, I'm not a communist (being preemptive because Americans, who are the majority on this site, tend to label anything that's not neoliberal as communist 🙄).
  • "If Microsoft's products are meant to be used by everyone then..." Well, they're not. They're 'meant' to be used by Americans only. Anyone else seems a fudge/mistake.
  • They could as well let it recognize non official expressions and such, read lips to improve understanding, recognize other languages. That would help much more.
    Such recognition could also help in controlling windows with just AI and simple camera.
  • Why would a Microsoft supported solution be must have, when there are already 3rd party solutions available?
  • Lets not get ahead of ourselves, cortana cant even do a layup like "what time is sunset".
  • Never thought to ask that and responded perfectly, to my current location, on my Windows Mobile, Canadian region HP elite x3! Lol
  • Unfortunately sign language isn't universal, so this would be another US only Cortana feature. Personally I would rather see support for more countries first. I've been a Microsoft supporter for many years, I have tried using Home Server, Media Center, Windows Mobile and Phone, Surface RT, Cortana and so on. Since switching to Android I've graudally moved away from all Microsoft consumer products. Google Assistant will support Swedish shortly, my TV Box is running Android TV, I have just bought an Google Home Speaker. All of these products would be gone from my home if Microsoft would release products that match the features offered AND made sure they weren't abandoned a year later.
  • Sign recognition, adding subtitles in AR... that'd be the future !