音声ブラウザご使用の方向け: SKIP NAVI GOTO NAVI

Web Posted on: August 4, 1998

Biosignals for Control of Graphic User Interfaces

Heide-Rose Vatterrott Sybille Hambach Oliver Hein Thomas Heider

Fraunhofer Institute for Computer Graphics
Rostock Branch
Joachim-Jungius-Straße 11
D-18055 Rostock, Germany
tel.: +49 381 4024 132
fax: +49 381 4024 199
email: vh@egd.igd.fhg.de

Summary

Interactive systems using information about eye movement and muscle contraction for control have been developed and tested in close co-operation between computer scientists at IGD Rostock and doctors of the Waldeck Clinic. With the help of these systems, it will be possible even for severely motor-disabled people to use computers.

Introduction

Computers can be used by disabled persons as aids in different situations, as tools for communication and environmental control as well as a source of information and entertainment. In addition, they are a vital tool for many disabled people in their integration into working life. One prerequisite of such a use is that disabled people are able to operate the computer input devices effectively [FS95], [GL96].

To use conventional computer input devices, people need to learn and perform special tasks. They have to be able to handle input devices designed for everyday work and normal physical conditions. The operation of such devices is not intuitive and impossible, or not easily possible, for persons with seriously restricted motor skills.

To give these people the chance to use a computer it is necessary to provide them with special input devices. One common way of developing new input devices is to modify conventional tools. Another method is based on the use of processed bio-signals as input information for human-computer interaction .

Bio-signals

Every person transmits regular biological signals. Some of them, e.g. eye movement, muscle contraction or speech, may be controlled consciously without training.

In order to use bio-signals as input information for human-computer interaction it is necessary to measure and analyse these signals and to extract information about significant actions.

Registering and Analysis of Bio-signals

The computer supported registering of biological signals is a well known aspect of current medical diagnostics. A great variety of appropriate devices has been developed during the last few years [WE92], [VÖ96].

The mode of operation of such devices is based on different methods, e.g. optical methods - registration and analysis of pictures - acoustic methods - registration and evaluation of sounds and bio-electrical methods - measurement and assessment of electrical potentials produced naturally in the body e.g. ECG, EEG, EMG, EOG. Methods useful for registering eye movement and muscle contraction are optical as well as bio-electrical ones.

The bio-electrical method for registration of eye movement exploits the fact that the eye, with retina and cornea, is like an electrical dipole. To record eye movements, two sensor pads are placed above and to the side of one eye to register changes of the electrical potentials. Sensors and amplifiers used for such a registration of eye movement can also be used to register muscle contraction. The technology for measuring voltages on the skin is very robust. However, the pads have to be placed at the appropriate points and connected to the gauges by means of cables. The use of bio-electrical gauges allows for precise detection of relative eye movements. It can be recorded whether the eye moved to the right, to the left, up or down. As far as muscle contractions are concerned it can be accuratly detected whether a muscle is tensed or relaxed.

There are many different optical methods for the detection of eye movement. One of them, detection with the help of an infrared sensitive camera, has been proved to be specially reliable and unaffected by surrounding light. Using this method the position of the centre of pupil and the position of the corneal reflection are detected and supply the basis for the calculation of the line of vision [LH96], [HR97]. In contrast to bio-electrical methods, this optical method can be used without sensors attached to the person and allows for detection of absolute eye movements. However, the resolution is limited particularly because of constant saccadic eye movement.

Bio-signals appear permanently, in contrast to signals appearing by use of conventional input devices. After recording biological signals by means of optical or bio-electrical gauges, the signals have to be analyzed for significance. Bio-signals are significantly different for each person and each session. For this reason, the gauges have to be calibrated before and sometimes while using the recorded signals for the control of interactive systems. By calibrating the gauges and adapting various parameters, it is possible to adjust the measuring devices to suit every single person for each session. The gauges may have to be re-calibrated during a session in order to compensate for signal variation caused by altered concentration or perspiration.

In using input devices based on bio-signal processing, it should be taken into account that the persons are capable of using these gauges to a widely differing extent. For example, some persons are unable to move their eye consciously either vertically or horizontally, others can move their eyes and in addition to this they can control some muscles. For application in the field of rehabilitation there are some additional requirements: The patients most not be additionally burdened by the input devices. The devices have to work reliably under different environmental conditions. The use of technology has to be financially viable.

Control of applications with help of Bio-signals

If the bio-signals have been measured using different methods and if significant user actions have been detected, it is possible to use the information gained for the control of various interactive systems. A distinction can be drawn between the control of newly developed special applications and the direct or indirect control of conventional applications, e.g. editors. Both kinds of applications have been developed at IGD Rostock.

By developing completely new applications, it is possible to adapt the interaction technique perfectly to the new input devices and to the symptoms of a group of patients or even of a single patient. Such applications are necessary for early rehabilitation as they can give the patients an early feeling of success. On the other hand, such applications are highly specialized and can only be used by a very small group of people. This disadvantage can be compensated for by providing some kind of authoring tool.

Two examples of such systems are shown in Figure 1. Both applications had been developed for a thirteen year old boy. This boy was born with an athetose. As a consequence, he has a serious motor disability and is unable to speak. Nevertheless he is able to understand and learn. Using computer programs controllable by eye movement he is able to solve mathematical tasks and to formulate texts in order to communicate with other people.

keybord

   
Figure 1: Training Program for naming images (left), Program to train simple mathematics (right)

In principle it is possible to use the software for analysing biological signals in the same way as a simple software driver. One example of this technique is the control of the cursor of a computer game by means of eye movements and the triggering of actions through muscle contractions. Using this technique, it is possible to control conventional applications.

Implementing and testing such solutions we found that they are often difficult to handle by the user, especially if the application program is slightly more complex and needs many different kinds of input information. Therefore we implemented a different solution - the indirect control of conventional applications. Using this method the software for analysing biological signals is connected to standard applications via a software program with a special user interface. One example of this category is a virtual keyboard to control a standard text editor.

Indirect control allows the user interface to be adapted to bio-signaling as an input device. This makes the software easier to learn and operate. Figure 2 shows two different virtual keyboards developed at IGD. The keyboard on the left can be controlled by means of a bio-electrical gauge. The letter currently focused on is highlighted and then selected by being looked at for the time set. Changing to neighbouring letters is carried out by looking to the left, to the right, up or down. Letters are arranged according to their frequency in the German language to facilitate fast typing even for non-proficient users. The keyboard is large enough to be visible for patients sitting in beds or wheelchairs. The keyboard on the right had been designed to be controlled by eye movement detected using optical methods. Both keyboards are able to communicate with the Microsoft standard editor Write.

keybord

   
Figure 2: Virtual keyboards as input devices for a standard editor

When using bio-signals as input information for human computer interaction it is necessary, as mentioned above, to design the user interfaces in accordance with the peculiarities of the method used for detection of bio-signals and the special skills of the user. The development of such applications is extremely expensive. If one is interested in using such solutions in the field of rehabilitation, it is necessary to find techniques for drastically reducing the implementation costs. Such techniques had been developed at IGD [HN98]. They are based on the concept of logical input devices and has been implemented using object oriented programming techniques.

Conclusion

The tools described in this paper have been developed in close co-operation between the Fraunhofer Institute for Computer Graphics, Rostock, and the Waldeck Clinic for the purpose of supporting the rehabilitation of severely motor-disabled people. Applying these tools to clinical practice, we found that the utilisation of bio-signals as aids to the control of interactive systems appears to be a promising alternative input technology.

Prerequisites for the application of this technology are: the patients have to be capable of understanding and co-operating and they have to be willing to practise using of the tools. People who trained with the system and experienced an increased quality of live have so far included: children with brain injury from early childhood, patients with locked-in syndrome and patients with muscular atrophy who are not able to speak.

Future work will include the development of additional applications as well as the utilisation of new devices for registering biological signals, e.g. speech or sound recognition, and the improvement of existing hard and software. In addition, we are looking for other application fields where we can apply our solutions.

Bibliography

[FS95] Fischbach, F.: Moderne Technologien und ihr Einsatz bei Schwerstbehinderten - Ein Interview mit W. Irresberger. Rehabilitation 34, 1995, S. 106-109
[GL96] Gill, J.: The Missing Links, Telecommunications for people with disabilities. EU, 1996
[GP96] Gips, J.; Olivieri, P.: Eagle Eyes: An Eye Control System für Persons with Disabilities. http://www.cs.bc.edu/gips/EagleEyes/EEReport.html
[HR97] Heider, T.: Einsatzvorbereitung eines Remote-Eye-Tracking-Systems für die computergestützte Kommunikation. Bericht RI 97 F 001, Rostock, 1997
[HN96] Hein, O.: Entwicklung von Interaktions- und Dialogtechniken auf Basis bioelektrischer Eingabegeräte, Entwicklung eines Kommunikationssystems für Schwerstbehinderte. Studienarbeit, Rostock, 1996
[HN98] Hein, O.: Softwarearchitekturen für adaptierbare interaktive Systeme mit alternativen Eingabetechnologien.- Diplomarbeit, Universität Rostock, Fachbereich Informatik, Rostock, 1998
[LH96] Lahoud, J.: The Eyegaze System. http://www.Ictinc.com/doc/optic.html
[VÖ96] Völter, S.: Virtual Reality in der Medizin I, Stand, Trends, Visionen. ISBN 3-9804428-0-2, 1996
[WE92] Webster Ed.: Medical Instrumentation, Application and Design. Houghton Mifflin Company, 1992