A Dutch version of QUEST (D-QUEST) applied as a routine follow-up within the service delivery process
R.D. Wessels, L.P.de Witte
R. Weiss-Lambrou
L. Demers
G. Wijlhuizen
iRv, POB 192
6430 AD Hoensbroek
The Netherlands
tel: + 31 45 5237640
fax:+ 31 45 5231550
e-mail: roelof.wessels@irv.nl
1. Introduction
The increased use of Assistive Technology (AT) and the continuous need for efficient and suitable service delivery are resulting in a growing demand for outcome assessment of AT. The users of AT, with their knowledge and personal experience, are a vital source of information; those who have to rely on the use of AT, have to consider it a viable solution to their problem. Outcome assessment enables those involved in service delivery to monitor the quality of their services and evaluate the money spent. In this context, user satisfaction can be considered as one of the main indicators of usability. An instrument that provides feedback on the satisfaction of AT-users with the outcome of AT-provision, is presently needed for clinical and research purposes. However, a reliable and validated instrument of that kind does not yet exist. In this paper we describe the development and the first field-tests of D-QUEST, a Dutch version of QUEST (the Quebec User Evaluation of Satisfaction with Assistive Technology) (Demers, 1996), as a standard part of the procedures of advising about AT and providing AT. We also describe the first results, focusing on feasibility aspects and reliability. This project was co-financed by the Dutch Foundation KBOH (Foundation of quality and usability research of technical aids).
In the Netherlands, mobility aids and home adaptations are provided by the municipality. A person who wants to apply for such kind of assistive technology device (ATD), is sent to an AT provision counsellor by his/her municipality. This counsellor assesses whether or not the person would be helped by the provision of AT and if so, what kind of ATD would be best. The counsellor forwards a recommendation to the municipality. The municipality then has to decide whether or not they are going to fund the provision and if they decide to do so, they contact a dealer and order the ATD. Finally the dealer delivers the ATD at the client's home.
| Top |
2. Objective
The objective of this project was to develop and field-test an instrument for assessing user satisfaction with AT provision. The instrument would have to be applicable as a routine follow-up within the service delivery process. It should be suitable for users of a wide variety of AT and with a wide variety of disabilities. The purpose of the instrument would be, to allow us to evaluate the result of service delivery in individual cases, in a structured and standardised manner. At the same time, this evaluation is meant to serve as follow-up service, enabling service delivery professionals to detect and solve problems, answer questions and correct imperfections that remain after the provision process has been completed. On a more general level, the instrument could potentially serve to monitor the quality of the service delivery system and its procedures.
| Top |
3. Methods
QUEST is an instrument currently under development by Demers, Weiss-Lambrou and Ska at the Université de Montréal in Canada (Demers, 1996), in collaboration with several Canadian and American institutes, the Danish Centre for Technical Aids for Education and Rehabilitation in Denmark and the iRv in the Netherlands. The current version of QUEST consists of 24 items, presumably influencing user satisfaction. AT-users are asked to score the relative importance and the degree of satisfaction with each of the items in relation to a specific ATD. This is done in a face-to-face interview, using 6-point scales and according to a specific procedure. Since QUEST is a generic instrument, not all items will be applicable to al kinds of AT. The respondent can decide which of the items he or she considers to be applicable or not applicable. The items are:
1. Simplicity of use Ease in using the ATD;
2. Repairs/servicing Ease in having the ATD repaired and serviced;
3. Maintenance Simplicity of upkeep and care of the ATD by oneself;
4. Installation Simplicity to assemble and/or set up the ATD;
5. Effectiveness Goal achievement with the ATD;
6. Cost Expenses associated with purchasing, maintaining and repairing the ATD;
7. Professional service Quality of information on ATD provided, accessibility and competence of professionals;
8. Durability Robustness and sturdiness of the ATD;
9. Multi-purposefulness Possibility to adapt and use the ATD for multiple activities/purposes; 10. Adjustments Simplicity in setting/fixing the components of the ATD;
11. Comfort Physical and psychological well-being associated with use of ATD;
12. Service delivery Ease in acquiring the ATD including length of time;
13. Follow-up services Ongoing support services for ATD;
14. Appearance Design, form, colour and acceptability of the ATD;
15. Transportation Convenience of transporting the ATD via the desired means of transportation;
16. Device compatibility Suitability of the ATD with other used technologies;
17. Weight Ease in lifting and/or moving the ATD;
18. Safety Degree to which the ATD is safe secure and harmless;
19. Dimensions Convenience of the device's size (height, width, length);
20. Motivation Incentive to use the ATD at home and in public;
21. Training Learning how to use the ATD;
22. Social circle support Support from family, peers and employer in using the ATD, whether physical or psychological;
23. Reaction of others Positive and encouraging attitude of others;
24. Effort Little physical or psychological exertion required in using the ATD.
The English version of QUEST (both questionnaire and manual) was translated by the first author into Dutch. Then this translation was back-translated into English by a professional translator with a background in occupational therapy. The differences between the original English version and the back-translation were discussed with the translator. (Guillemin, 1993) Finally, some minor adjustments were made, in co-operation with Dutch AT-provision counsellors, to adapt the instrument to the specific context of AT-provision and AT-use in the Netherlands.
During three months, the D-QUEST instrument was administered as a follow-up evaluation with 66 clients from the iRv service delivery centre, who had been provided with a wheelchair, an electrical scooter, a shower seat or a stairlift. Occupational therapists working as AT provision counsellors, applied the instrument, about three months after the clients had received their new ATD. As a method to reduce bias, counsellors did not interview their own clients, but only clients of colleagues. The instrument was also applied in another region by TNO-PG, a Dutch research institute. They administered the questionnaire (as part of a larger set of questionnaires) with a total of 306 people who had in the past received an AT provision (a wheelchair, an electrical scooter, a home adaptation or an adapted bed). In both regions respondents were visited at home. We asked the clients to score both their general satisfaction with the ATD and their satisfaction with each of the 24 items on a 6-point scale (with 1 = `not satisfied at all', 2 = `not much satisfied', 3 = `more or less satisfied', 4 = `quite satisfied', 5 = `very satisfied', 6 = `not applicable'). We used the data from the total group of 372 respondents (including both the iRv survey and the TNO-PG survey) to test the instrument for reliability and feasibility.
| Top |
4. Results
80% of the clients was very satisfied with their ATD provision in general. Although this is a high score, the fact remains that 19% is less than completely satisfied. The items that clients were most satisfied with were: `Durability', `Effectiveness', `Comfort' and `Dimensions' (76%, 76%, 76% and 71% of the clients respectively scoring `very satisfied'). The items that clients were least satisfied with were: `Service delivery', `Follow-up services', `Professional service' and `Effort' (15%, 11%, 11% and 10% of the clients respectively scoring `not satisfied at all', `not much satisfied' or `more or less satisfied'). A noticeable item is `Simplicity of use'. It scores relatively high on both high and low satisfaction (74% of the clients scoring `very satisfied' and 10% of the clients scoring `not satisfied at all', `not much satisfied' or `more or less satisfied').
None of the items turned out to be totally not applicable. The four items most often considered `not applicable' were: `Multi-purposefulness' (67% `not applicable'), `Transportation' (68% `not applicable'), `Weight' (68% `not applicable') and `Device compatibility' (75% `not applicable'). So still these items were considered relevant by 33%, 32%, 32% and 25% of the respondents respectively. We also analysed this per type of ATD. Our main groups were: manual driven wheelchairs (N=66), electrical scooters (N=50), shower seats (N=63) and toilet adaptations (N=100). We expected to find a different set of relevant items for each type of ATD. For manual driven wheelchairs and electrical scooters, we found the only item to be relatively irrelevant was `Device compatibility' (63% and 60% of the clients respectively scoring `not applicable'). For shower seats and toilet adaptations `Device compatibility' was also irrelevant (93% and 88% `not applicable'). Other items often considered irrelevant for shower seats and toilet adaptations were: `Multi-purposefulness' (81% and 90% `not applicable'), `Transportation' (89% and 94% `not applicable'), `Weight' (90% and 88% `not applicable'), `Training' (79 % and 82 % `not applicable') and `Adjustments' (74% and 80% `not applicable'). These finding were to be expected, since shower seats and toilet adaptations do not have to be moved or transported and their use is pretty unambiguous.
From some provisional tests we ran, we have reason to believe that the internal consistency is fairly high (Crombach's Alpha = 0.85 for the satisfaction answers and Crombach's Alpha = 0.79 for the importance answers). When we asked for any missing items, only few respondents mentioned one or two items. `Power / range of action' and `Ease of operation' were mentioned twice. A few other items were mentioned once. From this we concluded that we have captured pretty much the right set of items. We also ran some provisional tests to check the correlation between `importance' and `satisfaction' for each item. It showed that `importance' for none of the items had a very high correlation with `satisfaction' (Pearson correlation between 0.15 and 0.41). This suggests that distinction between these two aspects, `importance' and `satisfaction', is meaningful.
| Top |
5. Discussion
The first results give strong evidence of an enormous need with AT users for structural follow-up evaluation after AT provision. It turned out that a substantial part of the clients was stuck with problems or questions of some kind after completion of the service delivery process. These clients did not know who to turn to with these problems and questions. If they had not been visited by our interviewers, these problems and questions would have remained. In a lot of cases people with problems were not aware of probable solutions. They were not aware that a small modification to their ATD (like a change in the setting or the adding of a accessory) or to the environment (like some minor home adaptation) would solve their problem. In some cases it turned out that the wrong ATD had been provided. Since people, especially those who start using AT for the first time, often have little knowledge of the possibilities, they completely trust the judgement of the service delivery professional and decide they just have to learn to cope with imperfections. A lot of questions originated from insufficient information provided during the service delivery process. Of course all this should be prevented by improving the service delivery process. But this can only be achieved if a monitoring instrument is applied to detect these imperfections, like we did with D-QUEST.
Service delivery counsellors were very enthusiastic about the interviews. It gave them a possibility to visit clients at home and see how solutions turned out in reality. They noticed that problems occurred because of factors that were not taken into account (or could not have been taken into account) during the service delivery process. The structure of D-QUEST ensured that all possible problem areas were reviewed. The service delivery counsellors reported that they experienced the interviews to be very educational. Apart from this project, they have adopted the D-QUEST home visits as a part of the breaking-in of new counsellors.
In conclusion we can say that it is possible to measure user satisfaction with provided AT by using a generic instrument. The D-QUEST instrument turns out to be very useful and very well applicable both as a tool to evaluate individual cases and as an instrument to monitor the quality of the service delivery. The field-tests have given us a lot of clues for improvement of the instrument
| Top |
Literature:
Demers, Weiss-Lambrou, Ska: Development of the Quebec User Evaluation of Satisfaction with assistive Technology (QUEST) (Assist Technol 1996; 8: 3-13)
Guillemin, Bombardier, Beaton: Cross-cultural adaptation of health-related quality of live measures: Literature review and proposed guidelines (J Clin Epidemiol Vol. 46, No. 12, pp. 1417-1432, 1993).
| Top | | TIDE 98 Papers |