10. Analysis and evaluation

Contents - Previous - Next

10.1 Objectives

The objectives of this final chapter are to

• Discuss the types of analyses required in on-farm trials: technical, economic, and social.
• Review some of the characteristics of limited-resource families (i.e., households) and suggest what are likely to be suitable technology evaluation criteria for them.
• Describe some specific techniques for evaluating technologies with respect to technical feasibility, economic viability, and social acceptability.
• Discuss briefly some issues with respect to formulating recommendations, give an example of a specific technology in Botswana, and indicate one way to assess technology adoption.

10.2 Trial analysis: integrated analysis

Analysis of trials is the process by which team members evaluate and interpret trial results to determine the acceptability of a technology to farmers. Through the use of various analytical techniques, trial results are examined and evaluated systematically and used to predict whether farmers will find the technology acceptable, Of course, it is also important to try and ensure that this technology is potentially equitable in terms of its impact in the sense that its adoption by some families does not result in unfavourable impacts on the livelihood of other families in the:

• Short run -- for example, making it impossible for draught to be shared by different families.
• Long-run -- for example, resulting in ecological degradation.

In the long-run, only farmers, not plants or animals, adapt and adopt new technologies, Thus, FSD workers must learn to plan, view. and evaluate their work from the farmers' perspective -- to see the world through the farmers' eyes.

Analysis and interpretation take place after a trial is completed, but the planning for analysis must occur before the trial is implemented if maximum use is to be made of information obtained in the trial. The type of analysis planned is determined by the type of trial, but the implementation of the trial is also dependent on the type of analysis to be conducted.

The objective of analysis is to provide information for an integrated interpretation of trials, which can be used in planning further FSD, identifying relevant on-station research, and/or formulating recommendations for technologies prior to dissemination.

Analysis in FSD is performed on the data generated by on-farm trials, surveys, and studies. Once the data are collected and organized, the team can apply tools from different disciplines to generate a set of disciplinary analyses, which can be considered together in an interdisciplinary framework for final analysis and interpretation (i.e., integrated analysis),

Biological analysis of trials is usually the first step in analysing the actual trial data. At this step, the team determines if the new technology represents a significant biological improvement over the traditional system, The team must decide whether the results were obtained in a 'typical, environmental setting, and so are generally applicable, or whether they were produced under 'atypical' conditions (i.e., don't reflect farmer conditions) and must be interpreted with caution.

Of course, biological evaluation by itself is an inadequate indicator of whether the technology is suitable for dissemination to farmers. As a starting point for analysis, FSD workers seek to understand the household and farming environment. An understanding of the household and farming situation is important, because it serves as a basis for judging whether a technical change represents an improvement, To assess this, three types of analysis are important,

• Technological analysis is performed by the agronomist or animal scientist from the FSD team and determines if the new technology is practical in a technical sense. For example, can the farmer receive a greater yield or more animal product with the inputs and practices he/she can reasonably be expected to use with respect to the new technology?
• Economic analysis is used to determine if the farmer will receive a greater economic and more stable return from adopting the technology, Part of the economic evaluation is an assessment as to whether the farmer has enough resources available to adopt the technology or can acquire them by borrowing or receiving a government subsidy to facilitate adoption,
• Social analysis is used to determine if the technology is acceptable within the household (i.e., intra-household) and overall village (i.e., inter-household) situation. Socio/cultural analysis looks at the technology in a whole farm, analyzes acceptability for the various members of the household who are involved with the technology, determines if there are cultural factors that influence acceptability, examines consumption/nutrition implications, etc.

Another consideration in the social evaluation is an environmental impact assessment (i.e., whether there are likely to be long-term beneficial or harmful effects to using a new technology). Unfortunately, to date, much FSD work does not explicitly undertake an analysis of the environmental impacts of a technology. Researchers may do this subjectively, but it is important to do it explicitly. The whole area of long-term impacts of a technology on the environment is of interest to society and also possibly to the individual farmer. For example, the effect of a cropping system on the long-term fertility of the farmer's land, its impact in increasing or decreasing soil erosion, etc., are areas that deserve more consideration than currently they usually are given.

It is extremely important to all these types of analysis to evaluate whether a technology should be disseminated or not. For example, although biological interpretation may be made on the basis of a statistical test using a ().01 or 0.05 significance level, economic analysis may still be justified for some technologies that do not meet this level of biological significance. Two possible reasons for this include:

• Farmers may be willing to use a new technology, even when biological research results are not statistically significant. This is particularly true when the farmer's investment is small and the potential results are relatively large.
• Even though the level of biological return (i.e., yield) may not be significantly greater for a new technology, there may be other benefits. such as reduced labour demand at certain times, greater reliability in return, etc., that make the technology attractive to farmers.

All of the analyses should include and be tempered by farmers' reactions to the trial and researcher observations. The greater the degree of farmer management in a trial, the less quantitative scientific analysis is likely to be possible and the more qualitative farmer and researcher analysis will be involved. There is a continuum starting with RMRI trials, which have a Iarge amount of quantitative data for scientific analysis, and ending with an adoption study to determine farmer acceptance after the farmers have had a chance to adopt the technology on their own.

The final step is to bring all of the individual analyses together into an integrated interpretation of the trial results, The integrated interpretation of trial results probably should involve farmers and farm family members, FSD and on-station researchers, development/extension staff, and policy makers. Although it may not be possible to get all together in one group to discuss the interpretation of results, all have an interest in this interpretation and should be included at some point. Because different participants have different views of the benefits and costs from adopting a technology, there may not be a single all-inclusive interpretation of the results. An attempt to make an integrated interpretation, however limited, is of value to the FSD team in designing its future work plans, as well as giving the team members guidance in the question of likely adoption of the technology.

Numerous analytical tools are available to undertake the types of analysis just mentioned. The choice of tools depends on the purpose of the trial; the type of trial (RMRI, RMFI or FMFI); the data that have been collected, and the level of training of the researcher carrying out the analysis. Although many sophisticated analysis procedures are available to the researcher? they often have limitations concerning data or implementation of the procedure, which cannot be met under FSD conditions, and so are inapplicable.

The usual approach in practical FSD work is to use fairly simple procedures that are more adapted to field conditions and provide the basic information needed for interpreting trial results. Table 10.1 provides a listing of the primary tools that usually are used in the analysis process. Emphasis is on simple procedures that can be used under practical field conditions.

Even though FSD teams make use of all these types of analysis, they primarily rely on a two-pronged approach involving:

• Formal or Quantifiable (More Objective) Evaluation -- especially with reference to technical and economic aspects -- in order to provide convincing evidence to other researchers and planners, extension/development agency staff, and those responsible for formulating recommendations.
• Informal or Qualitative (Possibly More Subjective) Evaluation on the part of farmers testing it -- which could be based on a large number of criteria are weighted differently,

The following sections, do not discuss some of the standard statistical techniques listed in Table 10.1 that are used in station-based trial work (e.g., t-tests, analysis of variance, etc.) and survey work (e.g., frequency distributions, cross tabulations, chi-square tests, etc.), because they are well covered in standard statistical texts.

10.3 Suitable evaluation criteria -- farmer viewpoint

The ultimate success of a new farming technology depends on farmer evaluation, acceptance, adoption, and sustained use. Much of the success of the evaluation process hinges on selecting appropriate evaluation criteria. The difficulty of separating out these criteria is why FSD teams are increasingly advocating the two-pronged approach mentioned above (see Section 10.2), namely both formal (i.e., quantitative) and somewhat more informal (i.e., qualitative) approaches.

TABLE 10.1: ANALYTICAL TOOLS USUALLY USED IN FSD ANALYSIS

a. Descriptive statistics are common to several types of analysis for use in summarizing and describing data. The tools include: frequency distributions; measures of central tendency (mean, median and mode); measures of dispersion (standard deviation, variance, range, coefficient of variation); and graphics (histograms, bar charts, pie charts).

The primary issue is to ensure that effective evaluation takes place before technologies are recommended for dissemination by the extension service. Obviously, the best test of the value of a technology is the degree of farmer adoption that occurs. Unfortunately, such ex post monitoring is too late to prevent wasted investment in extension training and in policy/support system services developed to handle anticipated increases in production. Therefore, FSD teams try to evaluate the value of the technology ex ante or before it is disseminated widely to the mass of small farmers. A favourable assessment is essential if an official recommendation is to be developed, As implied to some extent in earlier discussion, there are three important points in determining the success of this ex ante evaluation.

• To the extent possible, the criteria used should be those that will be used by farmers in the target group (i.e., recommendation domain). As implied above, these are often difficult to identify.
• Farmer assessment is potentially an important complement to formal quantitative analysis.
• The balance between an evaluation based on the criteria used by local farmers and a societal evaluation -- such as unfavourable impacts on other families and on ecological stability -- is a difficult one to strike.

To help evaluate a technology from the farmer's point of view, some questions that might be useful to ask are:

• Is the problem to be solved important to the farmer? A problem may appear to be important from the researcher's perspective, but not from the farmer's.
• Do farmers understand the trials? Technologies building on small changes in the existing system probably are most easily understood.
• Do farmers have the time, inputs, and labour required for the improved technology? Farmers evaluate the resource requirements of a new technology not only in terms of whether they have or can secure the necessary resources, but also in terms of the competing demands for the resources. In addition, different family members control different resources and so may view the technology differently.
• Does the proposed technology make sense within the present farming system? A change in one part of the farming system generally causes changes in other parts of the system. How will these secondary changes impact the system as a whole?
• Is the mood in the region favourable for investing in new crop/livestock technologies? This may depend on how farmers view the economic and social situation in general. Is the proposed change compatible with the policy/support system that is likely to be available and with local preferences, beliefs, or community sanctions? Tastes and colour in foods, superstitions, the relation of the farming system to larger community values and activities all can effect the acceptance of a new technology.
• Do farmers believe the technology will continue to pay off in the long-term?

In a more formal sense, as was mentioned earlier (see Section 3.4), it is reasonable to assume that farmers will use criteria of two types in evaluating technologies:

• Whether they are able to adopt. Three factors important in determining this are:

• Is it technically feasible for the farmers to adopt the technology, that is, are the necessary inputs readily available and can all the necessary operations be undertaken technically'?
• Is the technology likely to be economically viable (i.e., profitable and dependable) for the farmers to adopt at the levels that will be beneficial to them? In this connection, it is important to be .sure that the proposed technology is more profitable and, hopefully, more dependable than the one it is designed to replace. In comparing technologies, it is important to determine that the proposed improvement increases the return to the most limiting factor. In a situation where labour or draught power are more limiting than land, it is important to look at the relative returns and dependability of those returns, in terms of the return per hour of labour or draught used during the most constraining period (e.g., the ploughing/planting bottleneck in assessing economic viability.
• Is the technology likely to be socially acceptable to the farming families? For some farming families who, for example share draught power, any proposed improved technologies that require more intensive use of draught power may not be acceptable because the potential for sharing may be decreased. As discussed above, there is also an issue of what happens to within household relationships.

• Whether they are willing to adopt. Broadly speaking, will farmers be better able to achieve their goals by using the improved technology'? As indicated earlier, the criteria on which they will decide their willingness to adopt often are difficult to determine. In addition to their experience in testing it and general feeling as to whether it helps in solving problems/constraints Important to them, other factors will include the balance between preferred foodstuffs and returns to labour during one particular period of the season. Such factors are particularly difficult to relate to for those who are 'conditioned' to an experimental method that deals basically in weight per unit area of land. Indeed, several such criteria may have to be brought to bear on a single proposed change. For example, staggered planting as a practice may indicate water, labour, or draught power scarcity, risk management, and a preference for a prolonged supply of a certain type of fresh food -- all at the same time. There is little chance of weighing these correctly in an economic analysis under the conditions in which many limited-resource farmers operate.

This difficulty in second guessing the balance in farmer evaluation criteria often has prompted FSD practitioners to argue for farmer assessment as a practical approach to technology evaluation. The FSD phase of technology development is in farmers' fields, with farmers on the spot. Yet, to date, the devices developed and used for farmer assessment are rudimentary, although, as discussed earlier (see Section 8.4.4), the use of some of the PRA techniques appears to have considerable potential. Farmer assessment is likely to be particularly useful when:

• Farmers have had little previous experience with the components of the new technology and, therefore, cannot be expected to provide useful information in an ex ante survey.
• Knowledge of a recommended technology is needed to focus the questions regarding farmers' experience, because there are many solutions to farmers' problems.
• The farming system is difficult to simulate in a formal model.
• Ex ante survey work has not been well conducted.

Improved methods for routine farmer assessment of technologies continue to need further development and refinement, Farmer assessment has been important in FSD work. Some FSD teams have made efforts to ensure that this took place through:

• Undertaking different types of trials reflecting various degrees of commitment on the part of farmers (see Section 9.2).
• Using farmer groups in assessing the values of the different types of technologies including work done at the RMRI level (see Section 9,8,6),

This emphasis on individual farmer's criteria for technology evaluation ignores the possible impact on other farmers and on broader societal issues, which is considered in a later section (see Section 10.6).

Contents - Previous - Next