Posted April 1999
Although extension educators, funders, and administrators are in favour of evaluating extension programmes, honesty requires us to admit that most of us are not overly enthusiastic about undertaking it. There are many reasons for our resistance. However, if we are to provide evaluation leadership, we must recognize forms of organizational and personal resistances, or these will sabotage our evaluation efforts. It is not uncommon to use the occasion of evaluation to mask our insecurity with distrust or criticism of others' intentions, motivations, competency, or adequacy of effort. We are sometimes caught in a bind between our own reluctance to ask ourselves hard evaluation questions and our resentment of outsiders asking questions.
The second form of resistance is fear, particularly the fear of change that evaluation might precipitate. It is the nature of organizations to be self-protective and defensive. Evaluation and organizational comfort appear to be somewhat incompatible because evaluation challenges organizations to change. We all resist change to some extent. Although evaluation is not revolutionary, it is a handmaiden to gradual change, and we have to recognize that our reluctance to participate in evaluation is partially a reluctance to embrace needed change.
Another resistance to evaluation comes from our need to avoid embarrassment about potential bad news. We need to know the quality of our efforts, but we also have a fear of finding out the truth about our achievements, particularly if we lack confidence. Most of us avoid tests for the same reason. We need to face up to our personal and organizational ambiguity regarding our need to know and our fear of knowing.
Still another real resistance results from the fact that evaluation is often an additional task to an already impossible workload. Those whose job description includes evaluation may need only to be reminded of this. However, potential benefits may have to be discussed and identified if collaboration is to become a reality among those who do not have a formal professional responsibility for under-taking it. Benefits include recognition for achievements, opportunities to improve practice, establishment of accountability, and learning new lessons about our efforts.
The term "evaluation" can be threatening. If evaluations are to be positively embraced by extension organizations, it may be necessary for those initiating them to be positive in the language they use to describe the proposed effort. Negative metaphors that are sometimes used for evaluation, such as "under the gun", "a mine field", "an invasion of inspectors", "last rites before hanging", and "defend the fortress" reflect past negative experiences with evaluation. Why not create more positive metaphors for the evaluation task? For instance, we could describe evaluation in medical metaphors such as "complete programme physical" to identify physical soundness and potential programme health risks. The term "auto-diagnostico" has been used in the Dominican Republic and "self-strengthening" in Sri Lanka. Or we could use a business metaphor, such as "taking stock" of profits and liabilities. We could use sports metaphors, such as "watching video tapes" of yesterday's game to improve our "game plan". We can use "mirrors" to see ourselves as others see us. We could consider evaluation to be a "learning process" or a "sabbath" day of reflection, renewal, gratitude, and celebration. Being self-conscious and deliberate about the language that is used regarding evaluation may help reduce unnecessary resistance to the effort and may build positive support.
Several evaluation myths have often discouraged extension managers from engaging in useful evaluation.
Myth #l: Evaluate only when mandated. Many funded programmes require evaluation as a form of accountability. However, it is a myth that evaluation should occur only if it is mandated. On the contrary, evaluations that are self-initiated are more likely to be taken seriously for immediate programme improvement. Programmes become responsible and excellent just as often through self-initiated evaluation.
Myth #2: Evaluation is an add-on. It is a myth that evaluation is an add-on activity or at most a pre-test with a post-test. It is most meaningful when it is integrated with decision making at every stage of programme planning and operation (Patton, 1991).
Myth #3: Evaluation is an activity for experts. It is a myth that evaluation should be undertaken only by technical experts. Yes, complex methods can be used; however, systematic evaluation can be undertaken by inexperienced managers, and specialists and educators themselves can be helped to critique their own work. This chapter is intended to help managers and specialists to do evaluation themselves.
Myth #4: Outside evaluators are best. It also is a myth that evaluation should be done only by external, outside, objective evaluators. Yes, external evaluators are often useful in challenging insiders to address what they have overlooked because of their nearsightedness. However, internal, self-initiated, and subjectively oriented evaluations also can be rigorous and valuable. In fact, because they often are participatory in generating, analysing, and interpreting data, they may result in greater acceptability of the findings and recommendations.
Myth #5: There is one best evaluation approach. Still another myth is that there is one best way to conduct an extension evaluation. Some approaches are probably better than others for addressing particular types of questions or concerns. However, the many types of evaluation approaches have their own strengths and limitations. Some situations require quantification and measurement, while others require qualitative, descriptive, and subjective data. Alternative approaches will be briefly described later in this chapter.
Myth #6: Quantitative data are best. A mixed-methods approach combining qualitative and quantitative methods can lead to better understanding and appreciation of phenomena under evaluation and provide triangulation, convergence, and corroboration of results from different methods. Qualitative methods are best for understanding the nature of something, while quantitative methods help in appreciating its extent. If we do not know the nature of something, we should conduct qualitative studies. After measuring something, we may still need to use qualitative methods to learn about variations and unique forms. The use of mixed methods can increase user responsiveness to evaluation information (Greene, Caracelli, & Graham, 1989). We should ask which combinations of methods are best for answering which evaluation questions, rather than deciding on a method and then forcing or changing our questions to fit that method.
This purpose of evaluation is sometimes called "formative" because the results are intended to help improve the programme during its formative stages. This is in contrast to "summative evaluations" when the purpose is to sum up or summarize the accomplishments at a point in time. When evaluations are to improve programmes, lessons learned about strengths and limitations of the programme are mined from the data so that changes can be made immediately. Sometimes the intent is to discover new approaches and alternatives or to adjust the programme to changing situations or client groups. Evaluation also is used to understand multiple reasons for apparent failure or to improve the management or operation of a programme.
Evaluation purposes tend to vary, depending upon where one stands within a system like extension. External funders often want an accountability purpose, while field staff are more likely to favour a programme improvement purpose. Policy makers and programme administrators can often appreciate an evaluation that contributes to new ways of thinking about extension or new forms of extension. Farmers want an evaluation to improve the benefits they may receive from extension staff.
Values, expectations, and interests are reflected in stakeholders' visions regarding what "good" looks like in extension. Some may favour an extension system that empowers the poorest of farmers in their struggle for more land and a larger share of market benefits. Multinational agribusinesses may view the success of
extension as increasing productivity for export. What a "good" or "excellent" extension system should look like will vary according to these different values and criteria. What different funders want may differ from what various groups of farmers may want in an extension system. Assumptions of stakeholders also vary regarding the relationship of extension education to larger social and political visions. For some, the extension system is intended to reproduce existing power relationships between government and farmers. For others the extension system is viewed as an instrument to transform those relationship. For some, extension is primarily a technology transfer system from researchers to farmers. For others, extension is a communication network among farmers, researchers, credit institutions, market organizers, consumers, and government policy makers. These expectations, values, and interests are reflected in the criteria that various stakeholders put forth as central for judging extension, regardless of the stated objectives in programme documents.
| External evaluator External criteria | Internal evaluator External criteria |
| External evaluator Internal criteria | Internal evaluator Internal criteria |
Politics is about power and power relationships. Education, including extension education, also is about power, because one way of increasing power is through increasing knowledge. Extension is political in that it can either reproduce existing access to knowledge or privilege or provide a redistribution of power in a society through increasing knowledge access. Probably the most crucial political question that evaluation can raise is, "Who benefits most from extension programmes?" This question is political, because the answer often reflects class and economic privilege, ethnic or racial dominance, and gender interests and relationships between staff and various sectors of the population of farmers. How does the extension system privilege some and not others? Where are extension staff located in relationship to the political, economic, social, and cultural structures of a country? How is extension affected by relationships that governments have to various sectors of agriculture? These values, expectations, and interests provide the background for negotiating the purpose, focus, and design, as well as the interpretation of the findings of evaluations.
What are several alternative approaches to programme evaluation? Choosing among these approaches is important because for each there are different assumptions about what data to collect, how to collect them, and how to make judgements about success. The following seven major approaches will provide a sufficient choice for most extension evaluation situations: (1) expert model, (2) goal-free model, (3) attainment of objectives model, (4) management decision model, (5) naturalistic model, (6) experimental model, and (7) participatory evaluation model.
Goal-free model. This approach assumes that outside evaluators do not know, or need to know, what the programme has intended to accomplish, but that it is the task of the evaluators to uncover what is actually happening relative to farmers' interests regardless of stated goals and intentions. The focus point is to identify environmental and farming conditions and then to compare these needs with what people are actually experiencing as a result of the extension programme. The gap is then viewed as a starting point for making changes in the programme. An example is an evaluation that describes conditions of indigenous farming groups cultivating fragile hillside soils and comparing these conditions with access to and appropriate content of knowledge from existing extension services. This approach relies heavily on open-ended interviewing and observation by persons who do not have a vested interest in the programme (Scriven, 1972).
Attainment of objectives model. This approach assumes that the success of a programme can be determined by measuring a programme's outcomes against its own goals and objectives. This type of evaluation begins with clarifying measurable objectives and then gathering data that validate the extent to which these objectives have been met. For this model to be credible, an essential feature should be added, namely, the evaluation of the appropriateness of goals and objectives, given the circumstances and needs of farmers. For example, an extension system may have adequately met its objectives of increasing production of maize among large landholders, but at the same time it may have neglected to question its lack of commitment to small landholders or tenant farmers. If an attainment of objectives evaluation is anticipated, programmes are often tempted to set goals quite low so that outcomes will be met easily, thus appearing to be successful while ignoring major challenges. This model also has a "black box" limitation in that it tends to ignore the extension process, thereby failing to provide explanations for outcomes (Provus, 1971).
Management decision model. The purpose of this model is to provide relevant information as a management tool to decision makers. It assumes that evaluation should be geared to decisions during programme initiation and operation stages to make results more relevant at each particular stage. Participation of stakeholders is central to the process because evaluation should serve their decisions. Sometimes cost effectiveness and operations monitoring are included (Stufflebeam, 1971; Tripodi, Fellin, & Epstein, 1971; Gold, 1988). One limitation of this model is the tendency for the decisions of major stakeholders to be viewed as more important than those of various types of farmers, especially women in agriculture who may not benefit directly from such an evaluation unless care is taken.
Naturalistic model. This model assumes that a programme is a natural experiment and that the purpose of evaluation is to understand how the programme is operating in its natural environment. There is an assumption that programmes are negotiated realities among the significant stakeholders and that evaluation serves this value-laden negotiation (Cronbach, 1981; Guba & Lincoln, 1989). Data should be collected and analysed from multiple perspectives. The outcome of the evaluation is dialogue concerning disagreements about objectives, expectations, problems, opportunities, policies, procedures, and suggested changes in methods or activities. Many positive collaborative changes can be made through this model
of evaluation if conflict resolution skills are combined with evaluation. Another purpose of this model is to diagnose or to identify the causes for certain behaviour on the part of some farmers, agency staff, or other development actors (Murphy & Marchant, 1988).
Experimental model. The purpose of this approach is to determine whether changes in programme outcomes (learning accomplishments) were due to the contributions of the programme and not just to life's experiences or from other influences (Goldstein, 1986). This model asks the question, "Were differences in sustainable agriculture practice attributable to the programme?" The simplest way to determine causality between the programme inputs and comparable groups, a group that received the educational treatment and a group that did not. This means that programme accessibility, at least during the experiment, programme consequences is to compare at least two is withheld from those learners who serve as a control group. Because of the nature of human subjects, the ethics of withholding educational services, and the difficulty of controlling for external influences, it is extremely difficult and costly to operationalize this model. It is recommended that this model be used only when major changes are expected or when a major failure is anticipated in pilot efforts where causal claims are central to making major programme investments (Rossi & Freeman, 1982).
Participatory evaluation model. The purpose of this model is for extension educators and farmers themselves to initiate a critical reflection process focussed on their own activities. This is done through identifying a persistent major situation such as extension's neglect of women in agriculture; subject it to critical reflection, underlying assumptions, habits of mind, and cause and effect expectations; and then after creating new assumptions, change practices and validate or invalidate the results. The model assumes a democratic participatory process along with autonomy on the part of educators and learners at the local level (Brunner & Guzman, 1989; Greene, 1988). This is a form of what is usually called "participatory action research".
Why use participatory approaches to evaluation? Involving people who are on the receiving end is likely to assure the most efficient allocation of scarce resources and the early identification of ineffective or wasteful use of resources. People on the receiving end are ultimately the best judges of impact, whether benefits have been produced or not (Uphoff, 1989, 1992). Being included in planning, implementation, and evaluation will show farmers that they are regarded as responsible, capable individuals and not simply passive "beneficiaries" or a "target group". Participatory evaluation is self-educating. It can encourage the development of human capacities among farmers. Without participation in evaluation, sustainable agriculture is unlikely. Participatory evaluation also can decrease the paternalistic, directive, impatient, or insensitive relationships among officials, technicians, and farmers by improving staff attitudes toward working with persons having less status and education. Although an initial cost of time is required, participation in evaluation can speed implementation when participants take greater ownership of efforts.
|
Level 5: Farmers conduct their own evaluation of extension independently of extension and report their findings to policy makers. Level 4: Farmers carry out evaluation of extension in cooperation with extension managers and make decisions regarding changes in providing extension services. Level 3: Farmers receive evaluation results and other information from extension staff and are asked to give reactions and recommendations for improving extension processes and resources. Level 2: Farmers receive information, evaluation summaries, feedback on extension performance from extension staff, but are not asked to react. Level 1: Farmers provide data and evidence of their achievements along with their reactions to extension without being involved in planning evaluation efforts. Adapted from Arnstein (1969) |
How much participation should the farmers have in evaluating extension programmes and their own learning? Farmers, in many extension programmes, have participated in evaluation of extension through farmer associations and committees. However, as Arnstein (1969) has pointed out in a much quoted article regarding a ladder of participation, there are various levels of involvement and participation. In her model, the bottom rungs provide for merely token participation, while the top rungs allow for participant control. A ladder adapted from Arnstein that provides levels of farmer participation in evaluation of extension is depicted in Figure 2. The practice of levels 3 through 5 will increase authenticity of data, reduce paternalism in relationships, assure relevance, and make possible a collaborative commitment to positive change.
Levels 1 and 2 can be characterized as pseudo participation because they represent paternalism on the part of extension. Levels 3 and above can be characterized as genuine participation because they represent collaborative or empowering relationships.
Farmers are by nature involved in informal evaluation of their own practice. What is at issue is the degree to which extension managers will deliberately collaborate and share control with them in collecting, analysing, and reflecting on evidence of their achievements and judgements in regard to the helpfulness of events, technologies, inputs, processes, and learning experience.
To Evaluating extension programmes, Part 2
 
|