Here’s How Voice Technology Is Becoming More Inclusive

Speakers offer captions, but speech-to-text options are growing

SpeakSee is developing a portable kit of smart microphones that transcribe speech to text in real time.

In 1972, Julia Child’s PBS show The French Chef became the first program to offer captions for the hearing impaired. It was a watershed moment for the deaf and hard of hearing in the U.S., and television became increasingly inclusive thereafter.

Almost 50 years later, captioning has made its way to smart devices from Amazon, Google, Microsoft and Apple. And while technologies have radically changed consumer behavior, captioning remains the primary accessibility option offered by big tech for the deaf and hard of hearing. Generally, when it comes to voice-enabled devices without screens, the hearing impaired are out of luck.

But now startups are emerging with voice technology of their own to try to bridge the gap with the hearing world.

Computer programmer Abhishek Singh, for example, is using machine learning methods like deep learning to make a screenless Echo respond to sign language. A camera interprets queries in sign language, which then are converted to text and speech. Alexa’s response is then transcribed on a screen.

According to accessibility startup Ava, captioning, subtitles and sign language add a visual layer to increase accessibility, particularly in group conversations where reading lips is harder and hearing aids are not sufficient.

Ava has an app to support in-person conversations by converting group conversations into text for the deaf and for the hearing impaired. The company did not respond to requests for comment, but according to Google Play, Ava has more than 50,000 downloads.

“Interest from the deaf and hard of hearing [in Ava] was very, very big. Many people were interested. There were tens of thousands of downloads, but only fraction kept using the app,” said Jari Hazelebach, co-founder of the startup SpeakSee and former business developer at Ava—and the son of deaf parents.

After conducting interviews with Ava users, Hazelebach said they pointed to barriers, such as all participants in a given conversation having to download the app, but also the stigma associated with hearing loss, which sometimes makes it hard to ask others to download an app just to have a conversation.

“Many people still associate hearing loss with oldness and don’t want to admit they are not able to hear what’s being said,” Hazelebach said. “It can be very exhausting having to be so proactive and trying to listen and lip-read.”

That’s in part what inspired Hazelebach to develop hardware a deaf person could hand out to start conversing right away. He said he posed the idea to Ava, but it’s a very big leap from software to hardware—hardware is generally expensive—and it didn’t work out.

Instead, Hazelebach spent 18 months working on a prototype with technical co-founder Marcel van der Ven. The two met on Founder2Be, which Hazelbach described as being like a dating site for entrepreneurs.

SpeakSee opened preorders in June for a portable kit of smart microphones that transcribe speech to text in real time. Sets include a charging dock and wearable microphones that capture the voice of the speaker, isolate it from background noise and transcribe the conversation to a smartphone. That means users can distribute wearable microphones prior to a conversation and follow along on their phones.

“It all started three years ago when my father, at the dinner table, was talking about ways to be included in conversations at work because he was not able to follow along in business meetings and missed out on a lot of relevant information to do his work successfully,” Hazelebach said. “Then we thought of speech recognition—technology to convert the spoken word to text instantly in real time, which is like a utopia for someone who is deaf—it’s easy to follow without having to do lip-reading or needing an interpreter, so that’s when we started thinking about speech recognition.”

Recommended videos