What you need to know
- A new post from Microsoft breaks down how Project Tokyo helps blind people recognize faces.
- The project uses artificial intelligence and specialized hardware to help send cues and information to blind people.
- The post focuses on a 12-year old boy, Theo, and the growth of Project Tokyo.
Microsoft continues to use technology to improve accessibility. A new blog post from Microsoft (opens in new tab) shares the growth and development of Project Tokyo, a combination of technologies that extend people's capabilities. Project Tokyo uses artificial intelligence, specialized hardware such as a custom HoloLens, and the results of several research efforts to help improve accessibility. One of the key areas that the post highlights is the ability to send spatial audio cues and information to blind people and people with low vision.
The post focuses heavily on the story of Theo, a 12-year old boy who uses a customized HoloLens to improve his communication. The headband detects people's faces, and if a face is on the system, sends an audio cue to Theo to tell who who he's facing. The post explains that some blind people can miss cues that people with full vision take for granted, such as knowing when someone is making eye contact with them. The system Theo uses helps communicate those cues and gives Theo and others an improved sense of connection.
Theo states that "It is so exciting to be able to find out where the people are in my environment," adding, "Not just who chooses to talk, but all of the people who are silent that you can see by their face, but I can't."
Project Tokyo isn't restricted to vision-related aspects of communication. Researcher Cecily Morrison views the future of AI as a set of resources that people can use in a variety of ways, "All of a sudden we don't have to say, 'Hey you are blind and I just made this accessible to you.' We say, 'Hey, you are you and I have just built a system that works for you.'"
Project Tokyo came from a challenge in 2016 to create AI systems that could do more than simple tasks. The blog post shares the story of how a team from Microsoft followed athletes on a trip to the 2016 Paralympic Games in Rio de Janeiro, Brazil. The trip helped Microsoft's team understand communication and group dynamics affected by blindness and vision impairments.
Ed Cutrell, a senior principal researcher at Microsoft, states, "We, as humans, have this very, very nuanced and elaborate sense of social understanding of how to interact with people – getting a sense of who is in the room, what are they doing, what is their relationship to me, how do I understand if they are relevant for me or not... And for blind people a lot of the cues that we take for granted just go away."
People working on Project Tokyo realized that the HoloLens already picks up a lot of the information that they needed in a system. It has a wide field of view, can recognize faces, and has speakers that support spatial audio. These features helped make a system that can identify people, contextually draw people's attention to a location with spatial audio, and to deliver cues to improve communication.
In addition to improving communication directly, Microsoft is looking into if Project Tokyo can help teach blind children or children with low vision to improve interactions.
Microsoft plans to expand Project Tokyo in the future. Its goal is to be able to adapt the system to different situations and preferences.
Sean Endicott is the news writer for Windows Central. If it runs Windows, is made by Microsoft, or has anything to do with either, he's on it. Sean's been with Windows Central since 2017 and is also our resident app expert. If you have a news tip or an app to review, hit him up at email@example.com.
Get the best of Windows Central in in your inbox, every day!
Thank you for signing up to Windows Central. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.