Technology for Students with Hearing Loss Has Wider Applications
How does a university student who is profoundly deaf or just hard of hearing cope with lectures where the spoken word is the main, if not the only, communication mechanism? Until recently, only sub-optimal solutions were available.
6 Reading the lips of the lecturer is not really an option as the student needs to be very close and the lecturer must always be facing them.
6 Having an interpreter attend all the lectures and either sign or lip-speak is expensive, difficult to organize as there are a limited number of interpreters and it makes the student dependent on a third party. This is only practical for major lectures or conferences where many Deaf people may be attending.
6 Using a stenographer to type the lecture in real time has the same limitations as an interpreter but has the benefit of providing a permanent transcript that can be used later.
6 Having the lecturer provide notes beforehand is not natural for the lecturer, nor will it reflect the dynamic nature of a live lecture.
None of these solutions are practical for an average student with hearing loss. In 1999, some universities along with multinational corporation IBM set up the Liberated Learning Consortium to see how technology could improve the situation. IBM’s participation was prompted by its new speech recognition product called ViaScribe™. The consortium set out to determine if it could be used in the lecture environment.
ViaScribe Trials
Some initial trials showed that the technology had potential but there were some major issues the first being that the lecturer was not adding in any punctuation ViaScribe was designed for dictation and the person dictating would add in commands like comma full stop new paragraph so the student ended up with text that looks like this paragraph which is very difficult to read in real time even worse it would pick up commands such as save and close and close down the application in the middle of the lecture.
Keeping the same speech recognition engine...
Some modifications were made...
First to stop the system from recognizing and acting on commands...
Second to recognize that lecturers speak in small chunks with pauses in between...
And laying out these chunks on separate lines with ellipses at the end...
Made the text much easier to read...
Various options were tried and tests are still going on but this format seems to work well...
As you can see from this example.
The technology proved itself in the real-time environment but it was not yet practical because of the difficult initial training and set up required of the lecturer. For the technology truly to be a solution, ViaScribe needs a rate of word recognition in the high 90th percentile. This is difficult to achieve and can only happen if the speech recognition engine can learn the lecturer’s voice. Professors are busy people and might be willing to spend up to a couple of hours training the system, but factoring in special vocabulary of particular academic fields, this was not normally enough.
The solution was to record the audio of the lecture, let the engine do its best and then have an editor correct its mistakes, thus “teaching” the speech recognition what it was missing. Equipment set-up at the beginning of the lecture was automated, reducing an unnecessary burden on the lecturer. After a few live lectures, the recognition rate dramatically improved into the 90s.
This means that a transcript of the lecture now can be provided in real time and displayed on a screen in the lecture hall. It is also possible to transmit it to a student’s portable device, such as a PDA. The student with hearing loss is now able to follow the lecture, just like any other student. With this technology, the student is no longer disabled and, in fact, may have an advantage. How often, when listening to a lecture, has a student wanted an instant replay of the last few sentences, either because he did not fully understand the first time or because he was momentarily distracted and did not hear everything? The technology display shows the last few chunks of information, thereby making the student using the technology more able.
Even more importantly, at the end of the lecture there is a complete transcript available on-demand. This is not just a boon for students with hearing loss but for all students. The ability to go back to the lecture and read the transcript of any section can only improve the learning experience.
ViaScribe technology has been developed much further by enabling the transcript of the lecture to be synchronized with audio or video recordings and with any PowerPoint presentation material. Over the Web and on-demand, a student can then see the transcript alongside the appropriate audio, video or slide. This is a complete solution for students with hearing loss that also provides completely new opportunities to other students, such as:
4 “virtually attending” a lecture
4 distance learning
4 better comprehension for foreign students who often find it easier to read than listen
4 for students with learning disabilities, the ability to replay and to choose the media
4 and for all students, the ability to have a search engine find relevant sections of the transcripts.
IBM ViaScribe has, up to this point, been developed with just the university student and lecture environment in mind. Now that it is functional in that setting, developers are considering other beneficial applications. Some research has started to look into the school environment with some initial success and some new challenges. However, findings of the research into ViaScribe’s use in pubic sector organizations and private enterprises suggest it will have enormous potential.
A study at RBC Financial Group identified the following opportunities:
Interacting with Clients
The RBC employee, trained in the use of ViaScribe, can speak naturally to the client who is deaf or hard of hearing (D/HH), or for whom English is a second language. The software simultaneously transcribes the conversation, making it available as a text display, and creating a text copy for the client. The synchronous audio and text transcript can also be used as a training tool to study employee-customer interactions.
Training and Other Classroom Applications
Multimedia lecture/presentation notes for in-house training and accreditation sessions can be made available via the Web. Real-time captioning of the presentation materials is also possible. Using ViaScribe in this way creates access for D/HH participants and creates an additional learning channel for non-English speakers.
Enhancing Teleconference
Real-time captioning of calls can eliminate comprehension problems commonly associated with poor audio quality during teleconferences. A complete transcript verifies the content for participants in real time, enhancing record keeping, and can be used to generate meeting minutes and allow full access to the content for those who are late or unable to attend.
Transcribing Webcast
Existing webcasts could be captioned, complete with speaker identification.
All these benefits are coming soon to a classroom near you. ViaScribe is not yet available as a commercial product but any organization that believes they could benefit from this technology should contact the Liberated Learning Consortium at www.liberatedlearning.com.
