Web Posted on: January 7, 1999

MAVIS - Mathematics Accessible to Visually Impaired Students Mathematical Gestalt Rendering Through Nonverbal Cues: An Audio Accessible Equation Browser

Christopher Weaver
New Mexico State University
Department of Mathematical Sciences MSC 3MB
P.O. BOX 30001 Las Cruces NM 88003
chrweave@nmsu.edu
Arthur Karshmer
José Enrique Méndez Mateos
Sandra Geiger
Gopal Gupta
Gabriel Lampert
Haifeng Guo
Daniel Grieb
Kelly Burma

Motivation:

A parse tree is a data structure which represents a hierarchical structure. Since mathematics is an inherently hierarchical structure, a parse tree representation of mathematical objects is natural. However, print notation provides much more than just the parse tree. It, in fact, provides a look up the trunk of the parse tree. That is, in one glance one sees a clear map or gestalt of the entire hierarchy of a given mathematical structure.

On the other hand, blind students have traditionally had mathematics presented to them using a markup representation. While this presentation is effective for recording mathematics in braille or audio it lacks the gestalt of traditional notation. This paper describes our development of an audio equation browser that delivers an audio rendering of mathematics using nonverbal cues and partial readings of deeper objects to convey gestalt. While we do not intend to replace existing tactile codes, we hope that this will make reading mathematics directly off of a computer easier. In the future we will incorporate this browser into an editor that will allow average blind students to create properly formatted mathematics independently.

Design:

The Equation Browser is a Java program that accepts a parse tree which is derived from a document written in LaTeX or MathML, the two most common computer-math scripting languages. In the case of math, the parse tree treats the outer most mathematical structure as the root of the tree and descending embeddings as its branches. For example, the parse tree for two-thirds appears as the following:

• Root-fraction;
• branch one, numerator-number two;
• branch two, denominator-number three.

Structured scripting languages like LaTeX or MathML lend themselves naturally to computers transforming documents written in them into parse trees. The problem is that the majority of humans do not grasp raw parse trees easily. This problem is heightened when the parse tree is delivered using only a sequence of verbal sounds. The Java program which we are developing takes a parse tree and examines it at the root level. The reader of a mathematical structure is given a tour of the branches using a musical note to identify each branch. Computer synthesizers are capable of generating musical notes with a number of different timbres. These different timbres together with the pitch values can be used to distinguish exactly where a reader is located in a mathematical structure. After a complete tour of the mathematical structure at a given level, the tones are then cycled through again allowing a reader to select which object to view more closely.

For example: the fraction two-thirds could be rendered by the following tone scheme.

• A trumpet blast at c4 indicates root-fraction.
• An organ pipe sound at g4 indicates branch one, numerator-number two.
• An organ pipe sound at g3 indicates branch two, denominator-number three.

After the tone scheme has been cycled through, it would be repeated unless the user decided to view one of the highlighted objects more closely. At the same time, the program reads parts of the structure being highlighted by the musical tones. It is possible for a reader just to have the major components of the highlighted structure read. It is also possible to have more detailed readings.

A more complicated fraction could be rendered in the following way.

• A trumpet blast at c4 indicates root, fraction.
• An organ pipe sound at g4 indicates branch one, numerator. "index three root" is read verbally.
• An organ pipe sound at g3 indicates branch two, denominator. "index four root" is read verbally.

Hitting the space bar to enter the numerator would cause the following sequence to be read.

• A banjo twang at c5 indicates object root, radical.
• A harp twang at g5 indicates branch one, index three. "index three" is read verbally.
• A harp twang at c5 indicates branch two-radical contents. "3x sup 2" is read verbally.

A similar sequence would be read for the denominator which also contains a radical with an index. More details about the contents of the radicals could be read at the time they are announced in the reading of the fraction. The level of detail given would depend on the user's preference. With the exception of tabular structures, the rest of mathematics notation is similarly hierarchical, and consequently can be read using a similar scheme. Tabular material can be rendered using a tone scheme that allows a user to pinpoint a particular item in a table.

Testing:

Testing will initially be conducted on blindfolded sighted subjects along with the few blind subjects working with our project. Subjects will be asked to read and compose mathematics with the system. During composition tests, subjects will be asked to both copy given materials and write mathematical expressions as they are computed. This will establish basic usability of this technology. It will also allow us to make coarse adjustments to the features of the program. Similar testing will also be conducted on blind volunteers nationwide to refine the program and to provide final statistical validation of the program.

The MAVIS project:

Mathematics Accessible to Visually Impaired Students is a project funded by the National Science Foundation at New Mexico State University. We focus our research on developing tools that make mathematical communication between blind students and sighted teachers easier. These activities include:

• an audio mathematics browser;
• a print math to Nemeth Code converter;
• a Nemeth Code to print math converter; and
• testing of accessible equipment developed at other sites.

For more information on any of these projects please see our web site.
www.nmsu.edu/~mavis