Web Posted on: March 3,1998

NEW SOFTWARE MAKES EYETRACKING VIABLE: YOU CAN CONTROL COMPUTERS WITH YOUR EYES

Gregory Edwards
Archimedes Project
CSLI, Stanford University
Stanford, CA 94305-4115
(650) 725-1725
gedwards@arch.stanford.edu

INTRODUCTION

Applications that are eye-aware can adjust to a user's eye-movements and can be controlled by a user with his or her eyes. The Black Squirrel Eye Interpretation Engine, an eye-aware software development tool that we are developing, recognizes multiple eye-movement behavior patterns and allows applications that make use of the tool to adapt and react to changes in a user's intentions. Many different types of applications can benefit from using eyetracking and the Eye Interpretation Engine. We will describe how the Engine works and give examples of applications that can be used by people with disabilities. A detailed description of how the Eye Interpretation Engine functions and how it was developed can be found in [1].

The Eye Interpretation Engine was developed out of our need to create eyetracking software for a person who has late-stage ALS (Lou Gehrig's disease). This person, who could move no part of his body except his eyes, needed to be able to fully control the keyboard and mouse of a computer so that he could maintain access to the Internet and all the software that runs on a typical computer. We set out to solve this access problem by creating an eye-activated on-screen keyboard so that the user could, in effect, type with his eyes into whatever Windows95 application he wished to use.

As we developed the eye-activated on-screen keyboard, our goal was to create software that would recognize and adjust to the individual characteristics of a user's eye-movements. In particular, we wanted the computer to adapt to the human user's eye movements instead of the human having to learn to control his or her eye movements in order to adapt to the computer. Additionally, we felt that the eye-activated keyboard had to be fast and effective for experienced users, but easy to use for first time users--allowing them to search for letters at a leisurely pace without worry that a letter would be mistakenly selected should they look at it too long--and we wanted the system to automatically handle both types of users. As we began developing the software, we were quickly confronted with the enormously difficult task of processing, filtering, and analyzing the huge amount of eye-position data returned from the eyetracking devices we used.

Making sense of a user's eye-movements is a difficult task without the proper tools. The software development tools that are available to interpret a user's eye-movements are limited to recognizing fixations, saccades, and blinks--the most basic interpretations of what the user's eyes are doing.

In the process of writing our on-screen keyboard, we created a high-level software development tool, which we call an Eye Interpretation Engine. The Eye Interpretation Engine greatly reduces the difficulty involved in writing eye-aware applications because it handles all the necessary low-level computations and allows the eye-aware application to focus on high-level issues. The Eye Interpretation Engine can recognize individual eye-movement behaviors, such as when a user is reading, when the user wants to select a button by eye, when a user starts and stops searching for a button, or when a user is looking around the screen with no intention of selecting any button at all. The power of being able to recognize the different behaviors is that it enables software written with the tool to adapt to the changing needs and expectations of the user.

NATURAL MOVEMENT OF THE EYE

People are not generally aware of how their eyes move and collect visual information. For instance, most people are not aware that their eyes typically move at least twice a second, jumping quickly in a straight line from looking at one location to another. These eye-movements are called saccades. People are also not generally aware that while they are searching for something, like a letter on a keyboard, their eyes pause up to four times each second, momentarily fixating on a key before moving on. It is during these fixations that the eye perceives the keys on a keyboard, the features of a person's face, or any other object in the world. Jacob [3] gives a good, detailed overview of how the human eye works and moves.

Though people are typically unaware of how their eyes move, eye-movements are a rich source of information about a person, which humans are adept at interpreting. Every day, people tailor how they explain subjects to others based on the listener's eye-movements and body language. It is also not too difficult to judge when another person is searching for something, again, based on observing the person's eye-movements.

CONTINUAL ADAPTATION TO THE USER

The software Eye Interpretation Engine we created interprets the natural eye-movements of users in real-time, and attempts to categorize the behavior the user is currently engaged in. Currently, we have defined three behaviors related to the use of an on-screen keyboard (or any application that has buttons that can be selected by eye) that can be recognized by patterns of eye-movement. The Eye Interpretation Engine can recognize:

• when the user starts and stops searching for a target, such as a letter on an on-screen keyboard;
• when the user knows where a desired target is located; and
• when the user is looking around the computer screen with no intention of selecting anything.

Additionally, we are developing and testing the Eye Interpretation Engine's ability to recognize different activities involved in reading. For instance, the Engine should be able recognize the difference between reading smoothly line-by-line, rereading part of a line of text, and referencing back into a paragraph by a user as the user tries to make sense of a complex idea. More advanced interpretations of reading and other movements are in development.

EXAMPLE APPLICATIONS THAT CAN MAKE USE OF THE TOOL

Software applications that make use of the Eye Interpretation Engine can adjust how they react to the user. This section will list just a few of the many different applications that can benefit from adapting to user's eye-movements.

On-screen Keyboards:

An on-screen keyboard is an excellent example of an eye-aware application that can greatly benefit from a system that adjusts to changes in the user's behavior. New users require a great deal of searching since they are not familiar with the keyboard layout--the system needs to recognize this and slow down to the user's speed. As the user begins to learn the location of commonly used letters, the user moves directly to those letters when she wants to select them--when the user does this it is only polite that the system not make her wait. Finally, as the user becomes more and more expert, the system should speed up to match her expectations--until the moment when the user needs to search for a little-used letter, which the system needs to allow by slowing down and letting the user search.

Since the Eye Interpretation Engine recognizes when a person starts and stops searching and it automatically adjusts to the characteristics of the user's eye-movements, eye-aware on-screen keyboards can be very easy to create, and very smooth to use.

Mouse Controller:

In addition to needing an eye-aware on-screen keyboard, the user needs to be able to use his eyes to move the mouse and click the mouse buttons. Using the Eye Interpretation Engine, Mouse Controllers that provide the user with full clicking and dragging functionality are easy to write. We have created such a program, which works with our on-screen keyboard to provide full control of any Windows95 program.

Communication Tool in hospitals for people who cannot speak:

Hospitals can provide an eye-aware communication program to people who have lost their ability to move and speak, either temporarily or permanently through a traumatic accident. Since applications written with the Eye Interpretation Engine automatically adjust to the user's eye-movement characteristics, the user can use the communication system without having to first learn how to control his or her eye-movements. The system's automatic adjustment to the user is also important because it means that technicians are not required in order to set up and tailor the system to each user.

Reading Aid for Children:

The Eye Interpretation Engine's ability to recognize when a person is reading, as well as its ability to recognize when a person has fixated on a word longer than normal, can be used to create a highly individualized reading aid for children. A word that a child is struggling with can be spoken by the computer; or the entire text can be spoken by the computer, keeping pace with the child as each word is read. Reading diagnostic software can be created that reports a child's progress to a teacher or parent, enabling the adult to focus on the individual needs of the child.

Reading Research:

Reading diagnostic tools or research applications can be developed that take advantage of the Eye Interpretation Engine's ability to recognize different activities involved in reading. The interpretations provided by the Engine can help a researcher filter through the enormous amount of data that a normal reading session can generate.

Cause and Effect Teaching Program:

Different patterns of eye-movement can trigger different events to occur on a computer screen. Children who need to learn the concept of Cause and Effect can use a program that displays pictures, animations, and special effects dependent on the child's eye-movements.

Screen-magnifier:

Each area where a user looks can be zoomed, effectively supplying a magnifying glass that follows the user's eye-movements. Alternately, areas of the computer screen can be selectively magnified if the user has difficulty only with certain regions of the display. By using the Eye Interpretation Engine's ability to recognize when a person has fixated for longer than is normal for that individual, small details on the screen can be magnified if a user is having an especially difficult time perceiving them.

Interface Analysis:

Interfaces for new programs can be tested, showing developers where users have difficulties while using the program. Users can try out different interface options and their eye-movements can be recorded and analyzed. Descriptive text in help systems can be tested by users, and developers can know how much of the text was read, as well as what areas of text caused confusion in the user.

Internet Usage Research:

Companies are currently spending huge amounts of money to establish a presence on the Web. Despite the money being poured into developing for the Web, designing information and advertising for the Web is not well understood. Eyetracking can offer a more accurate picture of what users do on-line, and, in particular, the Eye Interpretation Engine can help capture and describe the rich interaction that users experience. By knowing when a user is reading or searching for answers, and knowing what caused these behaviors, a better understanding of how people interact with the Internet can be gained.

PROGRAMMER'S INTERFACE

The Eye Interpretation Engine is implemented as a 32-bit DLL that currently runs under Windows95. The Engine automatically connects to and reads data from the various eyetracking devices. An application interacts with the Engine by linking to the DLL.

Integrating the Eye Interpretation Engine into an application can be very straightforward. All Windows programmers are familiar with event-driven programming, so an application can be written that simply waits for the Engine to report where the user wants to make a selection.

More advanced features of the Eye Interpretation Engine can be taken advantage of by software applications that want to harness the full power of an eye-aware interface. Multiple levels of interpretation of what the user is doing, or has done, are available at all times.

An external viewer is included with the Eye Interpretation Engine, which can run either on the same machine or on a second computer. The viewer enables the real-time observation of the entire history of all the eye-movements and various levels of interpretation, as well as a trace of the fixations and saccades as the user makes them. We believe the viewer will help many developers and researchers. The viewer has proven to be invaluable in helping us develop the Eye Interpretation Engine, recognize new patterns, and catch errors in our logic.

We have gotten good results when using the Engine with two different eyetracking systems: the Eyegaze system, from LC Technologies [4], and QuickGlance, from EyeTech Digital Systems [2]. The Eye Interpretation Engine is designed to work with any eyetracking hardware, and expanding the number of systems we support requires minimal effort.

CONCLUSIONS

Currently, it is far too difficult for the average programmer to write eye-aware software applications. In order to create an eye-aware application, programmers must be intimately familiar with how the eye works and with data filtering methods. Our aim is to integrate the knowledge derived from research into tools that programmers and researchers can use to quickly create eye-aware applications.

The Eye Interpretation Engine has a number of unique features. It can recognize natural eye-movement behaviors and, thereby, infer what the user's intentions are in specific situations. It can also determine from a user's eye-movements when each individual user wants to select a button by eye. These abilities enable software that uses the Engine to automatically customize themselves to each user's needs.

We believe that this is the only system that includes these abilities, and we have provided them in a software tool that others can use immediately. Bundling these capabilities into a developer's tool facilitates the creation of eye-aware software and increases the number of people who can contribute to the eyetracking field. We believe this will ultimately result in better, cheaper products for people with disabilities.

ACKNOWLEDGMENTS

This work was supported by a grant from the Packard Foundation and by reserve funds from the Center for the Study of Language and Information (CSLI), Stanford University. The first eyetracker used for research was purchased by privately donated funds. A patent is pending for the Eye Interpretation Engine, which is being funded by Stanford University's Office of Technology Licensing (OTL). OTL is also funding the commercialization effort. Any interested party may contact the author or Stanford's OTL at (650) 723-0651.

REFERENCES

[1] Edwards, Gregory, A Tool for Creating Eye-aware Applications that Adapt to Changes in User Behavior, ASSETS '98 conference proceedings. Also available on-line at www-csli.stanford.edu/~gedwards [2] EyeTech Digital Systems, www.EyeTechDS.com

[3] Jacob, R.J.K., Eye tracking in advanced interface design, in W.Barfield and T.Furness (eds.), Advanced interface design and virtual environments. Oxford University Press, Oxford, 1995.

[4] LC Technologies Inc., www.Eyegaze.com