Web Posted on: August 24, 1998
Improving Access to Computers for Blind and Visually Impaired Persons
- The Development of a Test Method for Usability
Stig Becker, Dan Lundman
The Swedish Handicap Institute
P.O. Box 510, SE-162 15 Vällingby, Sweden
Tel: +46 8 620 17 00, Fax: +46 8 739 21 52,
email: Stig.Becker@hi.se
1. Summary
Designers and developers of screen readers and other access tools, users of assistive technology, purchasers and rehabilitation experts all need relevant and up-to-date information on strengths and weaknesses of products on the market. Usability tests can provide them with needed information. Usability tests will improve product quality and promote the development of better products.
| Top |
2. Introduction
Graphical user interfaces (GUIs) are powerful tools for accessing computer systems. But not for everyone. The rapid change from text-based interfaces (DOS) into graphics (Windows) and the frequent use of colours, icons and other visual elements has created barriers for accessibility for users of Braille and synthetic speech.
| Top |
3. The CESAR Project
A number of technical solutions exist that build on translating the graphical user interface into synthetic speech or Braille. Although these access tools are developing all the time, they still seem to have limited functionality or have functions that are hard for the average user to understand, learn and use. Many users report that their access tools show technical incompatibilities, unwanted limitations, or unstable performance. Thus, a blind user has less access to computer applications than sighted users.
This perceived discrepancy between user needs and product claims laid the basis for the CESAR Project "Comparative Evaluation of Screen Alternatives for Reading". CESAR is a joint project conducted by the Swedish Handicap Institute (HI) and the Swedish Employability Assessment Institutes. The project focuses on usability.
The overall objective of the project is to promote improved accessibility. Blind or visually impaired users should have equal access to computer applications as sighted users. The access tools should provide the user with an interface that permits him to work with effectiveness, efficiency and satisfaction.
To reach this goal, the access tools have to meet very high and sophisticated requirements. Examples of needs of high performance are:
- - ease of installation,
- - simple or automatic configurations that enables easy identification and manipulation of icons, tabs, push buttons, menu items and other graphical objects in different applications,
- - good support for sighted co-workers,
- - usable documentation.
The first phase of the CESAR project was a large-scale evaluation of access tools for the Windows and OS/2 environments. In the study, 19 screen reading software were thoroughly tested, and the results verified clearly that a blind user has to accept a rather restricted accessibility to graphical computer environments.
By measuring the usability of the access tools we could provide developers with feedback about their products' performance. At the same time our comparisons of performance in different tasks serve as a helping tool for purchasers and users, planning to buy a new access device.
| Top |
4. Features tested
We started the preparation of our evaluation by analysing the different tasks that an access tool should perform. The following list summarises the areas of user-based quality requirements that were addressed in our test procedure:
- installation and configuration of speech and/or Braille devices
- application configurations
- monitoring the screen
- selections
- navigation
- alternative cursors
- responsiveness
- support for sighted users
- documentation
- DOS sessions
- file management
- word processing
- spreadsheet
- terminal emulation
- Internet access
- stability
- uninstalling.
| Top |
5. Test method
Our next step was to apply a well-established test method for measuring software usability. We contracted a well-reputed company with many years of experience in software design and usability tests. We found immediately that commonly used test procedures did not at all work. They had to be completely redesigned to adapt to the features of the access tools that we wanted to explore. We never intended to find one "best-in-test", ultimate product with the best quality, since quality may vary with intended use and with the user's capabilities. With the redesigned test method it was possible to form a picture of each product's characteristic strengths and weaknesses.
| Top |
6. Use of international standards
The role of international standards for software quality is growing, and we commonly invoke two such standards which are now evolving: ISO 9126 and ISO 9241.
ISO 9126 - Software Quality is an international standard that provides six main characteristics of software quality, along with many sub-characteristics that can be important to one software domain or another. These six main characteristics are functionality, reliability, usability, efficiency, maintainability and portability and they were emphasized during our tests.
ISO 9241 - Ergonomic Requirements for Office Work with Visual Display Terminals defines usability as: "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."
ISO 9241 also defines seven principles as important for the designed evaluation of a user dialogue. These are - suitability for the task, self-descriptiveness, controllability, conformity with user expectations, error toleration, suitability for individualization and suitability for learning.
The dialogue principles should be applied taking into consideration user characteristics such as:
- attention span
- the limits of short-term memory
- learning behaviours
- level of work and system experience
- the user's internalized view of the underlying structure and purpose
- of the system with which the user will interact.
Disabled users are not directly addressed in the standard, but the context of use is sensitive enough to take into account, for example, the special needs, requirements, experience and expectations of visually impaired users. In terms of these well-established principles, computer
systems including screen alternatives should facilitate user interaction in the following ways:
Wherever they find themselves in the system, users must be able to form an overview of where they are and where they are able to go, and how to navigate to that point.
They must be able to receive feedback on their own actions, and the actions of the system.
Functions and procedures in the system should be modelled on the user's own logic, capabilities, experience and terminology.
Systems should not burden the user's memory or other cognitive capacities.
Systems should be forgiving; users should be able to recover from errors or false steps.
Users should have access to help information and documentation to solve problems.
The CESAR Project relied on these principles in determining how to test products. A good technical solution for giving visually impaired or blind users access to GUIs is not enough; the solution must be simple, easy to learn and use, and otherwise promote a user's effectiveness, efficiency and satisfaction.
| Top |
7. Test procedure
One of the difficulties when using general test procedures in our study was that the tests take longer time than with sighted users. Therefore, the practical part of the tests was arranged as follows:
Only three test pilots were selected. All of the 19 access tools were tested by just one user. All three test pilots were experienced computer users. Two of them were blind and were skilled Braille readers. One was visually impaired and needed a magnification level of at least four times. We believe that the quality of our results benefit a lot from using blind and visually impaired persons as test pilots. None of the users were permitted to take part in the test of an access tool he himself was familiar with.
The users were not trained in advance in the use of the product under test. The user was supposed to use on-line help or user manuals when in trouble. Each of the above mentioned areas of requirements were tested by the user. We defined a set of subtasks for every area, which the user had to perform step by step. The checklist containing all these steps included more than 500 subtasks. The test leader gave the user his mission one at a time, and kept minutes of the performed results.
We decided not to include advanced or complicated tasks in the test. We rather selected functions that normally are learned in a basic course in Windows-95, in MS Word or Excel.
The results were noted on the checklist step by step. The columns on the list are Subtask, Ratings, Comments. When the test was completed, the checklist could be used as a test protocol for the tested product.
The Ratings column was used in a very simple way:
For example, when testing if a task can be performed or not:
2 - The task can be easily performed,
1 - The task can be performed, but it is not entirely usable or acceptable,
0 - The task can not be performed, is not available or causes an error.
In case of a zero in the Ratings column, comments were always added describing what went wrong. The completed test protocols were sent to the manufacturers for comments. In most cases this was the start of an extensive dialogue between the project team and the development departments. The Comments field was very useful for the companies as an input source for improving their products. Technical results on a more general level were communicated to the computer industry, primarily to Microsoft and IBM.
| Top |
8. Results
As indicated above, our evaluation clearly showed that there still are many limitations in accessing the graphical interfaces. The most successful blind users are either clever computer hackers themselves or have easy access to frequent technical support. The average blind user, on the other hand, can not easily explore the screen and experiment with available objects like a sighted person normally does when in trouble. He has to rely on a sighted support person. Our usability tests pointed out these differences in behaviour. The advanced blind users often tend to underestimate the problems and weaknesses with his access device, because he can always find ways to get around them. Well designed usability tests can assist the manufacturer in focusing on the most important improvements. Our impression after the publication of the results is that there is a very positive interest from the companies in getting user feedback through this kind of systematical tests.
| Top |
9. Conclusions
Usability tests give very useful data which serve as a guide for product purchase. They are also an important source of information for the developers. User involvement and systematic usability tests will give the necessary know-how to make graphical user interfaces fully accessible to blind and visually impaired persons. However, test methods need further development. The CESAR usability study in 1996/97 was just a first step. In collaboration with other organisations we plan to continue our work on promoting accessibility by evaluating access tools and by providing the industry, the users and purchasers with relevant information.
| Top | | TIDE 98 Papers |