DEAF-BLIND USERS GRASP THE IDEA WITH DEXTER,A PROTOTYPE ROBOTIC FINGERSPELLINGERSPELLING HAND

Deborah Gilden, Ph.D.,

The Smith-Kettlewell Eye Research Institute

San Francisco

Brad Smallridge, Upstart Robots

ABSTRACT

The intelligibility of a new version of Dexter, a robotic fingerspelling hand, was tested in its ability to display random letters and unrelated sentences. All fFive deaf-blind subjects (who range widely in their ability to read tactile fingerspelling), were able to read all, or almost all, tested the intelligibility of Dexter, a robotic fingerspelling hand. Scores were perfect or near perfect for understanding both of the random letters and meaningful but unrelaanted the sentences displayed by Dexter. INo time limitations were imposed, however, and it was evident, however, that tactile reading ofusing the roboticmechanical hand iswas much slower than tactile reading ofreceiving fingerspelling from a human fingerspeller. In order to To speed up the process without losing intelligibilityincrease speed but not lose intelligibility, we will modify Dexter is being modified to some of the letter shapes as well as make Dexter's movements more similar to its dynamics closer to that of a human handfingerspeller. We will also add telecommunication In addition,display capabilities telecommunications capability will be incorporated in the system.

BACKGROUND

Technology has gone far in eliminating distance as a barrier to communication. Telephones, facsimile machines, and e-mail have made communicating with someone on the other side of the world almost as easy as communicating with someone in the same room. But bBecause use of these technologiesmedia requires vision or hearing for their use, they have not been generally accessible to mostnot had a similar impact on people who are both deaf and blind.

Deaf-blindness among adults is most commonly the result of a genetic condition called Usher's syndrome (1). Since the vision loss starts during the teen years, it takes over a decade to unfold , it is common for often mistaken for simple congenital deafness. The Usher's syndrome childis frequently educated in a special program for deaf youngsters, and receives no preparation for the vision loss which inevitably occurs later on. The student learns to read print and to communicate via either sign language (which includinges fingerspelling), or speech reading (a comprehensive expansion of lip reading). These are all vision-based skills.  

The gradual reduction of vision seen in Usher's syndrome typically occurs during the adolescent years. When the gradual vision loss becomes severe enough that extreme, it interferes with all aspects of communication. The individual who received special training to develop skills so useful to a deaf person can no longer the Usher's patient can no longer see the hands which fingerspell, the lips which speak, or the print which is displayed on paper or TDD (telecommunication device for the deaf). NewOnce again special skills mustneed to be be learned to compensate for further reduced sensory input.

The most efficient solution is to learn braille. For blind people braille serves as a medium for reading and for accessing computers; for deaf-blind people braille serves the additional role of communication tool. The most popular braille device which enhances the communication ability of deaf-blind braille readers is the TeleBraille. The TeleBraille is a TDD with a braille display, and can be used for either face-to-face communication or for telecommunication.

STATEMENT OF PROBLEM

Relatively few Usher's syndrome patients are able to take advantage of braille materials and displays. One reason is that most programs to teach braille are designed for, and available to, hearing children, not deaf adults. Also, there is typically great reluctance among those with Usher's syndrome to learn braille. Learning a new code designed for a less efficient modality is not appealing to individualsthose who are proficient in communicating via sign language and print. Learning to read Using braille also has important emotional implications. It is perceived as the ultimate "admission" of blindness -- a difficult step for those left with even a smalltiny amount of vision.

The non-visual communication systemof choice for those with severe vision loss from Usher's syndrome is usually becomes a modification of the familiar: a hands-on versionon version of sign language or of fingerspelling. Both methods have their drawbacks. Hands-on sign language is extremely awkward and fatiguing. It requires both hands of fatiguing as it requires both hands of the interpreter to be in constant contact with both hands of the deaf-blind person. It also requires a lot of arm movement in space without the clear body reference points used in visual sign language. In contrast to hands-on sign language, tactile fingerspelling requires only one hand of the deaf-blind person to be in contact with one hand of the interpreter, and each person's hands may remain stationary.   However, this method though it is is tedious and time-consuming as each word must be spelled, letter by letter.to spell each word, tactile fingerspelling is generally preferred over hands-on signing.

In addition to the constant physical contact demanded by hhands-on sign language and/or tactile fingerspelling, both have several other limitations as well. They restricts communication to the small number of individuals who know these codes -- and are willing to employ them with someone who crequires direct physical contactannot see. The constant physical contact is awkward, restricting, and fatiguing. Also, these communication systems do not give deaf-blind individuals access toare not helpful in accessing written materials, computer displays, or telecommunications.

METHOD

To address the aboveproblems, the RERC Rehabilitation Engineering Research Center at The Smith-Kettlewell Eye Research Institute has engaged in collaborative efforts to develop a mechanical fingerspelling hand named "Dexter." (2,3,4)1.. The hand, named Dexter, receives ASCII from an interfaced  in response to messages typed to the deaf-blind user. As Dexter receives these signals, it forms the appropriate letters of the American one-hand manual alphabet of the deaf. By feeling these hand configurations, deaf-blind individuals can "read" the messages being typed. In a similar fashion, Dexter can also provide the user with access to computer information.computer access.

The Smith-Kettlewell Institute is currently working with a San Francisco company, Upstart Robots in San Francisco, to develop a commercial version of Dexter which could be used for both for direct face-to-face communication and telecommunication (5). Toward this end Smith-Kettlewell and Upstart Robots recently developed a new Dexter and evaluated itsthe intelligibility of Dexter III.2 The testing was conducted with the assistance of five deaf-blind subjects whose tactile fingerspelling skills ranged from novice to expert. They were asked to "read" individual letters, as well as unrelatedsimple sentences, presentedsentences by which Dexter presented to them.. The letters were presented as alphabet sets, displayed randomly without replacement. The subjects were asked to read three such sets during one session. The subjects were allowed to determine the rate of letter presentation. No time limits were imposed. The sentences were also presented three times.

RESULTS

Initial intelligibility for the 26 letters of the English alphabet, presented in random order without replacement, ranged from 20 to 26 correct. Based on instant feedback, we made slight modifications to some of the finger shapes during sessions. By the third (final) presentation one subject correctly identified 23 letters, and four subjects correctly identified all 26 letters.

The confusion matrix in Figure 1 shows the subjects' responses for each of Dexter's 15 presentations of each letter (three complete sets for each of 5 subjects). A column total of more than 15 indicates that a subject gave more than one response; totals of fewer than 15 indicate at least one individual did not make a response. The table shows that only one error was made in the reading of letters B, C, G, I, J, L, R, S, and Z; two errors were made in reading letter A; five in reading P; and five in reading Q.

Good intelligibility was also seen in Dexter's presentations of ten unrelated simple sentences weresentences. also highly intelligible. In almost every instance the deaf-blind users were able to accurately recite all, or almost all, of the words in the sentences.

DISCUSSION

An analysis of the nature of the few letter errors gives useful insight regarding what is needed to improve the hand. For example, both P and Q are made with downward wrist flexion. While Dexter is capable of this movement, the pressure of some of the users' hands prevented it from completing this action. The fact that in four instances the letter Q was mistaken for G, is an example of this problem.

Sentence-reading errors seemed related to memory overload from the time needed to recognize letters displayed by an unfamiliar device, rather than to errors in identifying the letters. Receiving tactile information from a mechanical hand appeared to require much time and concentration. The increased time and effort of reading a mechanical hand When errors did occur, they often seemed to be due to is added to the heavy memory demand of fingerspelling in general: recognize each letter, retain the sequence of letters, blend these to form a word, retain the sequence of words, and finally blend the sequence of words to form the entire sentence.

FUTURE PLANS

A detailed analysis of the errors made by the five deaf-blind subjects, along with their comments about the device, are guiding the design of a new Dexter. The researchers will also modify certain dynamic aspects of Dexter's performance to make them more similar to those of a human interpreter. These improvements should make it faster and less demanding to read the hand, especially in real-life situations where meaningful sentences appear in context. In addition, Dexter's usefulness to the deaf-blind community will be expanded by adding telecommunications capability to the next generation of this system.

Letters Presented By Dexter

Figure 1. Confusion Matrix of Random Letters Read Tactually by Deaf-Blind

FUTURE PLANS

A detailed analysis of the number and types of errors made in the transmission of letters and sentences to the five deaf-blind subjects, the researchers' observations about the transmission process, and subjects' comments about the device, are guiding the anatomical and functional design of a new hand. The finger positions to form some of the letters will be slightly modified, and some of the dynamic aspects of Dexter's performance will be altered to make them more like those of a human interpreter. These improvements are expected to make reading the hand faster and less demanding, especially in real-life situations where relevant sentences appear in meaningful context. In addition, Dexter's usefulness to the deaf-blind community will be expanded by adding telecommunications capability to the next generation of this system.

REFERENCES

1. Davenport, Sandra L.H., "Usher {sic} Syndrome: Vision and Hearing Loss," Hereditary Deafness Newsletter of America, Boys Town National Research Hospital, vol. 4 (1), Summer, 1994, 1-5.

2. Gilden, D. and B. Smallridge, "Touching Reality: A Robotic Fingerspelling Hand for Deaf-Blind Persons," Proceedings, Conference on Virtual Reality and Persons with Disabilities, sponsored by California State University, Northridge, June 17-18, 1993, Millbrae, California 50-54..

3. Gilden, D. and D. Jaffe, "Dexter, A Robotic Hand Communication Aid for Deaf-Blind," International Journal of Rehabilitation Research, 1988, 11 (2), 188-189.

4. Gilden, D. and D. Jaffe, "Speaking in Hands," Soma Magazine, October 1987, Vol. 2, No. 3, cover plus 6-14.

5. Gilden, D., "Understanding Dexter: The Intelligibility of a Prototype Fingerspelling Hand," Proceedings, ECART 3, October 10-13, 1995, Lisbon, Portugal, sponsored by the European Conference on the Advancement of Rehabilitation Technology, 164-167.

ACKNOWLEDGMENTS

The author would like to thank Rustie Rothstein, Southwest Region Representative, Helen Keller National Center, for her input regarding Usher's syndrome.

The project described was supported by grant number 1R43 DC00890-01A3 from NIH, NIDCD. Additional support for the development of this device has come from DOE, NIDRR and The Smith-Kettlewell Eye Research Institute.

Deborah Gilden, Ph.D., Associate Director

Rehabilitation Engineering Research Center

The Smith-Kettlewell Eye Research Institute

2232 Webster Street

San Francisco, California 94115, USA

Telephone: (415) 561-1665

Fax: (415) 561-1610

E-mail: debby@skivs.ski.org

Mr. Brad Smallridge, President

Upstart Robots

933 Treat Avenue

San Francisco, CA 94110

USA

1 This concept originated at the SouthwWest Research Institute, San Antonio, Texas. Smith-Kettlewell's original collaborations to develop a modern hand were with students from the Department of mMechanical eEngineering students at Stanford University.

2 The project described was supported by grant number 1 R43 DC00890-01A3 from NIH, NIDCD. Additional support for the development of this device has come from DOE, NIDRR and The Smith-Kettlewell Eye Research Institute.

Deaf-Blind Grasp Idea With Dexter