DESIGNING EVALUATION OF CBR PROGRAMMES

SELF ANALYSIS BEFORE STARTING AN EVALUATION

1. What do you need to know and why ?
   • Relevance of `vision', `mission', its clarity and transparency( needs and resources analysis).
   • Efficiency of systems of management.
   • Short term effectiveness of interventions.
   • Long term impact of the interventions.
   • Sustainability of the financial, technical and personnel aspects of the programme.
   • Community ownership of the programme.
2. What are the sources of information and the relevant indicators that are available?
3. Who will benefit from analysing the problems and identifying the solutions ?
4. Who will make decisions based on the information from the evaluation ?
5. What resources are available for the evaluation ?

AREAS REQUIRED TO BE COVERED DURING EVALUATION

Sources of information selected for a study are those which get impacted by policy and practices in rehabilitation, and are required to be included in a study to increase its validity. By allowing the input of the "stake-holders" into the research process, issues of relevancy and applicability can be addressed more directly. However, it can be argued that with this introduction of non-researchers and qualitative parameters, the data becomes subjective, biased, and unreliable.

Cultural factors affect how others view disability and treat them; they also affect rehabilitation practices, and seem to be the reason why stake holder groups must be included in research.

HOW DOES A FACILITATOR CHANGE A CONVENTIONAL GROUP INTO A PARTICIPATORY GROUP ?

Characteristics of Conventional Groups

• Quick thinkers and articulate speakers utilise most of the time.
• Participants interrupt each other often and are not ready to hear another person's point of view.
• Differences of opinion amongst members are viewed as conflict that must either be stifled or solved.
• Expression of minority views are not allowed and conclusions of the majority or sometimes the vocal minority are thrust on the rest of the group.
• Questioning the speaker is often perceived as a challenge to him.
• A large number of non-speakers are not interested in the transactions, and participants usually rehearse their speech in their minds while others speak, rather than listen to the transactions.
• A large number of participants avoid speaking out on controversies because they are not too sure whether they are saying the things that the group wants to hear, and what is quoted as personal opinion by the speaker is often the prevailing public opinion because the speaker feels the threat of being left out of the group if he voices his personal opinion. Personal opinions are often voiced behind each others' backs and usually outside the venue.

Characteristics of Participatory Groups

• All members of the group participate, giving enough room for each other to express their views, even if they express opposing viewpoints.
• The group participates, paying attention to all the speakers, often clarifying the meaning of what they say, and listening intently to the proceedings.
• Participants speak their own minds rather than voice public opinions, without the threat of being disowned by the group.
• Participants disagree with each others' views amicably in the group, rather than speak behind each others' backs or outside the venue.
• Even if the group opposes the view of an individual participant, he makes it a point to express his convictions.
• Problems are solved not by stifling dissenting participants, but by convincing them through reasoning.

QUESTIONS TO BE ANSWERED BEFORE USING A TOOL

1. What information are you looking for ?
2. Is there a need for collecting this information ?
3. What questions need to be asked to collect this information ? Are they reliable and valid ?
4. Who will answer these questions and are they willing to answer them ?
5. Who will ask the questions ? Do they have the skills to do it ?
6. Who will analyse the inferences ?
7. Is this exercise conducted at an affordable cost ?

DIFFERENT EVALUATION METHODS

Surveys - Select questions, select variables as indicators to answer the questions, choose methods to collect the questions, choose appropriate sample, train data collectors, conduct a pilot test, collect the data, analyse data, interpret data.

Interviews - Structured, semi-structured, open-ended.

Individual, key informants, large groups, focus groups

Observation - Prepare check-list.

Participatory Rapid Assessment (PRA)-Qualitative, open-ended, group, interview method. Cost-effective, participatory, quick, comprehensive information.

USEFULNESS OF QUANTITATIVE DATA

1. Data are accurate.
2. Gives the broad picture of a large population.
3. Identifies major differences in trends of a population.
4. Useful for establishing baseline data.
5. Useful to statistically establish cause-effect relationship.

USEFULNESS OF QUALITATIVE DATA

1. Useful to identify amorphous data on change( qualitative variations of indicators).
2. Requires less time and money.
3. More effective than quantitative data with a skilled practitioner.

TIPS TO AVOID BIAS IN EVALUATION

1. Avoid leading questions.
2. Use representative sample.
3. Avoid powerful key informants.
4. Do not feel reluctant to ask uncomfortable questions.
5. Listen to every aspect of the answer rather than selective aspects.
6. Do not exhibit annoyance during interviews.
7. Avoid promoting positive answers.
8. Train evaluators to use the questionnaire in the same manner with different clients.
9. Collect same information from different groups.

STEPS TO BE CARRIED OUT IN ASSESSING NEEDS

1. What are the needs expressed by clients, their families and the community?
2. What are the priorities in the community
3. What are the existing beliefs and attitudes related to the intervention ?
4. How do different groups view the efficiency of existing services ?
5. How do different groups identify the lacunae in existing services ?
6. What changes do these groups feel are necessary ?

PURPOSE OF SHORT TERM AND MEDIUM TERM PROCESS EVALUATION

1. To describe the on-going process
2. To ensure that the resources are effectively used
3. To plan for efficient work
4. To identify problems early and to make the necessary changes
5. To identify potential opportunities
6. To design a structure for future impact evaluations

EPIDEMIOLOGICAL STUDY IN DISABILITY

Tools

The validity of the `Ten Questions' was tested in Bangladesh. It was found that TQ was very sensitive in identifying moderate and severe disabilities in the age group of 2-9 years. It has a high false positive rate, but low false negatives. Hence it is very sensitive, though not very specific. A positive result on the TQ has to be always followed by a more specific evaluation.

Sample size

1. Sample size should be planned when a study is planned.
2. `Sample size' is the approximate minimum number of persons that should be studied within a particular limit of cost and precision.
3. Formula to calculate the sample size:

   A=3.8416 PQW n (sample size) = A/(E2+(A/N))

P= Assumed population prevalence in %.
Q=100-P.
W= Likely design effect, which is about 1.5 in fairly simple sampling methods. Random sampling is generally with less design effect. Cluster random sampling gives more design effect, but is cheaper. Stratification of the population tends to reduce the design effect, but is quite cumbersome and does not give the advantage in cost and accuracy as one would expect.
E = Maximum acceptable random sampling error in %. This is usually set at 0.95 or 95% confidence limit, which means that the prevalence rate of a condition identified can be interpreted with 95% confidence. When prevalence is very small or one is dealing with a rare condition, error may be much more than the prevalence itself. Hence the error allowed will have to be lowered to a smaller acceptable level. In these instances the sample size will become quite large to get adequate precision.
N = Population size.

If a sample size is prefixed due to financial reasons, then it can be worked backwards, to identify at what confidence level it can be interpreted and whether it is a valid interpretation.

Dr. Maya Thomas & Dr. M J Thomas
J-124, Ushas Apts, 16th Main, 4th Block, Jayanagar, Bangalore - 560 011, India
Tel and fax : 91-80-6633762
Email : thomasmaya@hotmail.com

Printed at :
National Printing Press
580, K.R. Garden, Koramangala, Bangalore - 560 095 Tel : 080-5710658

Table of contents