SCOTT SIMON, HOST:
A new opera in Omaha not only presents disability as a theme, it presents disabled singers and actors on stage, front and center. Two of those cast members are nonverbal. Jeff Lunden reports that the production worked for five years to develop technology to give them voice.
JEFF LUNDEN, BYLINE: If you remember physicist Stephen Hawking, he used a special speech synthesizer to let him express himself. Jakob Jordan uses similar technology.
JAKOB JORDAN: It is great to be here as someone who was not able to fully communicate for the first 22 years of my life. It is mind-blowing to be in an opera and to be here sharing on NPR.
LUNDEN: But as part of that new opera, "Sensorium Ex," Jordan, now 23, uses technology developed by a lab at NYU that makes his voice less robotic, more expressive.
JORDAN: When I first heard the sound in my voice come to life, a new realization was born - dream the bigger dreams, you know, the ones you dismiss and hide away because they seem impossible.
LUKE DUBOIS: Even though it might still sound a little stilted, a little synthetic, it's not perfect, it's them, right? It sounds like them.
LUNDEN: Luke DuBois is co-director of the NYU Ability Project. He and a team of technology professors and grad students developed an AI interface to let Jakob Jordan and the castmate he alternates with, Kader Zioueche, speak in the opera.
DUBOIS: We had this, you know, pretty profound eureka moment. It was, like, very emotional in the room. Everybody was like, oh, my God, 'cause it no longer sounds like their off-the-shelf commercial speech synthesizer. It sounds like they're talking.
LUNDEN: This all came about because composer Paola Prestini was commissioned to write an opera and collaborated with librettist Brenda Shaughnessy, who has a nonverbal son.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED ACTOR: (As character, singing) [inaudible].
LUNDEN: They developed a dystopian sci-fi story and cast the piece with disabled performers. One singer has cerebral palsy, a couple of others are blind, and one character is nonverbal, says Prestini.
PAOLA PRESTINI: You know, I had a challenge. How do you bring a nonverbal character to life? You know, how do you bring a nonverbal character to life in an opera, where everything is about voice?
LUNDEN: Paola Prestini wanted the nonverbal character to not only be a stage presence, so she approached Luke DuBois - who, in addition to running a technology program at NYU, is a composer - and asked him to develop a device to allow this character to speak. In addition, Prestini says...
PRESTINI: And we both agreed that it would be open source so that by the time we finished the opera, that this would actually really be something that anybody in the nonverbal or minimally speaking community could use.
LUNDEN: It was a five-year process. The first two years were spent surveying the nonverbal community to find out what they wanted and needed from such a device.
PRESTINI: There's this mantra that, you know, everybody in this community uses, which is, nothing about us without us.
LUNDEN: And then DuBois and his team set about developing sophisticated technology that combines sampling of the sounds the two nonverbal actors make, using AI to give them expressive voice. They also created devices to allow the two of them to control those expressions onstage. DuBois says these gizmos...
DUBOIS: Are repurposed time-of-flight sensors - and sometimes you call it lidar. So these are the sensors that you have on your automobile that help you park or know when you're about to get hit by another car. They're collision detection modules. They're about three bucks.
LUNDEN: And by waving his hand over these little modules, DuBois demonstrates how Jakob Jordan can manipulate the prerecorded sound of his voice.
DUBOIS: Moving my hand over one of them will trigger it.
JORDAN: I am me.
DUBOIS: And then the other two...
JORDAN: I am me.
DUBOIS: ...Will let me slow it down...
JORDAN: I am me.
DUBOIS: ...And speed it up, right?
JORDAN: I am me.
LUNDEN: These devices can even generate some of Jordan's other sounds, whistles and hums.
DUBOIS: They can also do it with their own natural vocalizations, which we've sampled.
JORDAN: (Whistling).
DUBOIS: So, you know, Jakob can play and bend his whistles.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED ACTORS: (As characters, singing) Ah.
LUNDEN: Jakob Jordan says...
JORDAN: They took our sounds and incorporated them into our performances, so each showing will be a unique experience that reflects us. Thank you for hearing our voices.
LUNDEN: The opera premieres at the Common Sense Festival (ph) in Omaha, which focuses on arts, science and disability acceptance. And everyone involved is hoping the technology can be picked up and developed to provide voice to people who, for whatever reason, cannot speak. For NPR News, I'm Jeff Lunden.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED ACTORS: (As characters, singing) Ah. Transcript provided by NPR, Copyright NPR.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.
