Web Posted on: August 24, 1998

Evaluating Assistive Technology Prototypes: Laboratory or Real Life Contexts?

Anna-Liisa Salminen
STAKES P.O. BOX 220, Fin-00531, Helsinki, Finland
tel: +358 9 3967 2089, fax: +358 9 3067 2054
email: annaliisa.salminen@stakes.fi

Helen Petrie
Sensory Disabilities Research Unit
University of Hertfordshire,
Hatfield AL10 9AB, U.K.
tel: +44 1707 284629, fax: +44 1707 285059
email: h.l.petrie@herts.ac.uk

1. Summary

Usability evaluation of assistive technology (AT) is vital to ensure that products meet users' requirements and become successful in the marketplace. Typically, usability evaluations are conducted in artificial laboratory environments, but more contextual approaches such as constructive technology assessments, also have a growing interest. Laboratory evaluations are highly controllable whereas contextual evaluations are more realistic. Both approaches are necessary before commercialisation of products. Laboratory evaluations are important during early phases of the product development, in order to develop usable interfaces, but the prototype should not be produced without also evaluating it in real life contexts. This paper will outline approaches from technology assessment and usability evaluations that can contribute to appropriate development of AT products.

| Top |

2. Introduction

The participation of disabled and elderly users is now widely accepted in the design and development of assistive technology (AT). Different methods can be used to ensure user participation in this process. However, user participation as such does not guarantee that the outcome of the work will be good. Problems which often impede effective user involvement include the fact that users often find it difficult to envisage new types of AT or the approach that has been chosen for user involvement is not suitable for the particular situation.

This paper will outline approaches from technology assessment and human-computer interaction that can contribute to appropriate user involvement and evaluations of AT products. In doing so, we will compare the advantages and disadvantages of laboratory evaluations and contextual evaluations. These two approaches will be illustrated with examples from a number of TIDE Projects, including the ACCESS and VISTEL Projects.

| Top |

3. Evaluating assistive technology prototypes

The evaluation of AT at a prototype state is vital to ensure that end-products meet users' needs, are genuinely usable and are successful in the marketplace. Usability is a multi-dimensional concept which in the context of AT can be defined to include:

• effectiveness: how well does the AT allow the user to achieve the goals they wish to achieve;
• efficiency: how quickly does the AT allow the user to achieve the goals they wish to achieve, and with a tolerable error rate;
• learnability: how easy is it to learn how to use the AT (for technologies which are not used very frequently this may also include how easy it is to remember how to use the AT when the user returns to it),
• flexibility: how many different tasks, in what range of environments, does the AT work
• satisfaction: how satisfied are users with the AT (this may depend on the above factors, but may also relate to factors such as the social status/stigma attached to the technology) [8].

Evaluation of AT is an ongoing process which should occur during the different stages of the development process. Evaluations need to be situated within an iterative design lifecycle in which user requirements are elicited, a system is developed on the basis of these requirements and then subjected to usability evaluation [6]. The system may then be re-designed in light of the results of this initial evaluation, and the cycle of design and testing may be repeated several times. Neilsen and Sano [5] cite an example in which he went through 20 iterations of the design lifecycle in producing a set of icons for use on the SunWeb interface.

3.1 Laboratory evaluations

Laboratory usability testing is often conducted in highly artificial situations that may be very different from the kinds of environments in which most people live and use AT. Laboratory evaluations test a prototype or product in controlled settings, where the users are usually given a pre-defined set of tasks to perform. Methods in laboratory evaluations may be videotaped observations, log files, interviews, group discussions and questionnaires. The types of data collected can often be qualitative, but emphasis is often on producing quantitative results, which may require extensive coding of the data by the investigators. For example, measures such as the time spent in completing tasks, the number and types of errors, usage of a manual, and users' quantified opinions about the product can all be used as information to assess the usability of the product.

For example, in the VISTEL Project [4, 9], seven visually impaired people were asked to undertake a usability evaluation of a rapid prototype of an adapted version of a screenphone. They were each given an introductory session explaining the functionality of the phone and its interface and were allowed to practice using the phone. They were then asked to perform three tasks with the phone, which had been chosen to include most of the major functions of the phone. This evaluation took several days to plan, was conducted in one working day and the results were analysed in about a day. It provided a sufficient information to produce a set of guidelines for the designers to initiate work on the next prototype.

3.2 Contextual evaluations

In contextual evaluations, prototypes are evaluated in real life, in a setting which is similar to actual usage of the product. The usability of the prototype is seen as a multi-layer concept as the evaluation has additional dimensions beyond assessing the usability of the prototype, which are related to rehabilitative goals, physical and social environment as well as services. Methods in contextual evaluations are partially similar to laboratory evaluations, including observations, interviews, diaries, questionnaires and group discussions. The type of data collected can be both quantitative and qualitative, but the emphasis is on qualitative data.

AT is targeted to accomplish a number of activities in daily living: self care, study, work and recreation. These activities take place in specific contexts, and these contexts are additional requirements for the technology. Therefore, according to the constructive technology assessment approach, AT needs to be considered in its full social context, since its usability is as much dependent on this context and the material infrastructure, as of its technical effectiveness and efficiency. This social context includes not only end-users, but also other people such as caregivers, relatives, teachers, therapists, and other professionals. It also includes the service delivery system for AT, other social support systems and the environmental context [3].

In the ACCESS Project the final evaluation of the communication software for speech impaired people took place with users in real life contexts for six months. Seven users who represented different age and diagnostic groups and who lived in different environments participated the evaluation. The methods in the evaluation included observations, questionnaires, interviews, videoing, using a diary and using log files. In the evaluation, there were difficulties due to delays with both hardware and software. It was difficult and time-consuming to find suitable portable computers, to choose and source suitable input devices for the users and to build their personal symbol libraries. Also as the holiday season started in the middle of the installation phase of equipment for the evaluation, the project team had to try to reach the users either at their residential homes, permanent homes or at summer cottages in order to install their device, provide user support, maintain the device and then to conduct the evaluation.

However, the results of the evaluation were very fruitful. The new communication device did increase the communication and psycho-social skills of four users. Their motivation to communicate and their social status increased. Of the other three users, for one user it proved too difficult to design suitable personal selection set. The other two users preferred other communication devices. As a result of the evaluation, many technical improvements were made to the communication device and ideas for future development were noted [7].

3.3 Evaluations in laboratory or in real life contexts?

Evaluations in laboratory or real life contexts can be compared considering a number of issues including: user involvement, cost, reliability, validity and ethical issues.

Users are involved in both laboratory evaluations and contextual evaluations, but in contextual evaluations users have usually more control over the tasks they undertake for the evaluation. In addition, other user groups such as relatives, therapists and close associates also regularly participate in contextual evaluations; this is extremely rare in laboratory evaluations.

All evaluations need to be cost-effective. Laboratory evaluations often require expensive facilities. On the other hand, laboratory evaluations are usually much quicker to conduct than contextual evaluations (see our examples above), which is very important for rapid product development. Careful selection of tasks can reduce the time spent in the laboratory evaluation to a few days, although analysing the resulting data, especially video data, may be extremely time-consuming.

In contextual evaluations, there is no need for expensive equipment for evaluation, but each user needs their own equipment which may be expensive if they are individually built prototypes. Contextual evaluations are time-consuming, taking weeks or months depending on the prototype or product. Data analysis is also a slow process, especially if there is a lot of qualitative data to analyse. In addition, contextual evaluations cannot take place without staff with appropriate rehabilitative expertise.

The reliability of the evaluation is important. Well-designed laboratory evaluations produce highly reliable data as the evaluations can be repeated in exactly the same conditions. It is possible to manipulate different variables associated with a prototype and study their effects precisely. For example, if different interface styles are to be compared, users may be asked to undertake a series of tasks with the different interfaces which are carefully matched for difficulty. All other aspects of the situation are controlled, for example speed of response of the system, other hardware and software used or lighting conditions. Any differences in performance can then be reliably attributed to the difference in interface. In addition, laboratory evaluations produce exact measures to guide the further design of a product.

Contextual evaluations tend to have lower reliability. The validity of such evaluations depends on the purpose of the evaluation. Laboratory evaluations support development of a working prototype, but contextual evaluations support development of a working product by showing what really happens in the field. For example, laboratory evaluation of communication software may assess if users are able to select a symbol from a selection set, whereas in contextual evaluation additional aims of the evaluation may be to investigate whether a communication device can be used for discussions or if users increase their social circles or improve their discussion skills. Contextual evaluations are highly valid from rehabilitative point of view because they give feedback from those real situations which AT is intended to target. In addition, contextual evaluation raises service related problems which retailers may face and serves as a way for industrial partners to learn about the necessary product related services.

Ethical issues are important in both laboratory and contextual evaluations, but in different ways. In laboratory evaluation, the emphasis is on whether participants are asked to do anything which may have any adverse consequences for them, either immediately or in the longer term. This can range from the stress of being asked to do tasks which are actually impossible to complete, or the boredom of undertaking many repetitive tasks. The confidentiality and anonymity of the situation are also important, so that participants should feel free to comment about a product without worrying about the consequences [1,2]. In contextual evaluations, expectations for a prototype may be very high among users and they tend to expect services similar to those for real products. Difficulties in use and any delay in maintenance can cause frustration. Therefore the whole project team has to commit themselves to a contextual evaluation and take responsibility for providing a realistic system and service to the users. The prototype of the product must be good enough for contextual evaluation; users should not be given empty promises and they should not be left to struggle alone with a prototype. If problems with maintenance or user support are not dealt with properly, it may weaken the results of the evaluation, irrespective however of how good the device is technically.

Both laboratory and contextual evaluations are necessary when evaluating AT prototypes. Laboratory evaluation can be recommended especially in the early phases of the development process as it is quicker and good for detailed prototype development. However AT prototypes should never be commercialised before evaluating them in real life contexts. Contextual evaluation is especially important when the technology being evaluated is demanding and when the user is especially dependent on the new technology. Neither of these types of evaluations is inexpensive, but in the end good evaluation should prevent expensive mistakes in the marketplace.

| Top |

4. Conclusion

Any kind of usability evaluation is better than conducting none at all. Laboratory evaluations are highly controllable, but artificial, whereas contextual evaluations are more realistic, but less controllable. Both approaches complement each other and are necessary before commercialisation of a prototype. Laboratory evaluations are necessary during different phases of the product development, in order to develop usable interfaces, but the prototype should not be produced without evaluating it in real life contexts.

| Top |

5. References

[1] American Psychological Association. (1992). Ethical principles of psychologists and code of conduct. available at: http://www.apa.org/ethics/code.htm

[2] British Psychological Society. (1997). Code of conduct, ethical principles and guidelines. Leicester, U.K.: British Psychological Society.

[3] Cronberg, T. (1993). Technology in Society: a Search in the Seamless Web. Unpublished manuscript.

[4] Johnson, V., Petrie, H. and Mercinelli, M. (this volume). An investigation of the user needs for screen-based telephony for people with visual impairments.

[5] Neilsen, J. and Sano, D. (1994). SunWeb: User interface design for Sun Microsystem's internal web. Proc. 2nd World Wide Web Conference '94: Mosaic and the web (Chicago, IL, October 17 - 20), 547 - 557. Also available at:www.ncsa.uiuc.edu/SDG/IT94/Proceedings/HCI/nielsen/sunweb.htm

[6] Petrie, H. (1997). User-centred design and evaluation of adaptive and assistive technology for disabled and elderly users. Informationstechnik und Technische Informatik, 39(2), 7 - 12.

[7] Salminen, A-L., and Kaasinen, E. (1996). Contextual software evaluation with speech impaired users. Proc. HUSITA 4 Conference. Finland.

[8] Shackel, B. (1991). Usability - context, framework, definition, design and evaluation. In B. Shackel and S. Richardson (Eds), Human factors in informatics usability. Cambridge: Cambridge University Press.

[9] VISTEL Project home page, available at: http://paperino.cselt.stet.it/ufds/vistel/

| Top | | TIDE 98 Papers |