10.4.1 Assessing New Equipment Prototypes
Quantitative and qualitative techniques are used to evaluate technical handling and other aspects of equipment alternatives. Typically, open-ended qualitative survey techniques are used to evaluate how well the equipment handles, to identify issues, and to discover thoughts about changes that might be made. In some cases, these assessments are combined with physical measurements (e.g., efficiency of work indicators such as dynometer readings of pulling power and work rates for a draft animal harness, or precision of work indicators such as seed placement or seed damage rates for a planter) to complete this technical evaluation,
During assessment of the operational handling aspect of alternative equipment, strategies, etc., varied reasons for poor operational handling will be encountered, such as:
When handling problems can be foreseen clearly by FSD staff or farmer participants, it is usually advisable to make corrections or necessary adjustments before undertaking statistical testing. Assessments of operational handling may not always require a control for comparison, Controls would not be very meaningful if the experimental operation was qualitatively different from other conventional operations in the farming system.
Informal one-on-one exchanges or group discussions involving farmer participants, FSD staff, and relevant experts are highly effective and informative methods of assessing the handling characteristics of alternative practices. PRA techniques, such as matrix scoring (see Section 8.4.4), potentially are ideally suited to quantify these informal assessments.
Even when the initial assessments of handling performance are negative, significant progress still can be made if insights and suggestions lead to better designs and pinpoint needs for instruction and farmer training.
10.4.2 Assessing the Technical Responses of Alternative Technology Options
Standard statistical techniques listed in Table 10,1, as indicated earlier (see Section 10,2), are the same as those used in station-based trial work (e.g., t-tests, analysis of variance, etc.) and are well covered in standard statistical texts, Therefore, this section simply covers some of the issues involved in choosing which statistical procedures to use in on-farm research analysis,
A number of helpful observations and suggestions have been made by various authors about quantitative evaluation techniques that pertain to FSD in particular, though no single set of standards has emerged yet, The following represent some of these observations:
- Predictive technical analysis will most often be based on statistical regression and related statistical techniques. However, other options exist that may become more prominent in the future (see Section 10.4.4).
- Two important requirements for obtaining accurate predictions of technical responses are:
-- That the data represent a sufficiently broad spread of environments and of the factors that affect the response. This spread must cover the full range of circumstances for which conclusions are sought. It would be highly inappropriate, for example, to make general conclusions based on responses measured only in years with below average rainfall. To achieve this spread of circumstances, numerous authors have written that in FSD work, it will be more important to include a larger number of farms rather than devote FSD and farmer resources to replications within the farm or field (see also Sections 9.5.1 and 9.5.2).
-- That trial designs be streamlined, straightforward, and convenient, such that farmers are able to accurately implement and manage trials in a manner consistent with their own management style. In FSD, this is usually given as another argument for single replications per field or farm.
If FSD researchers discover interesting responses that were not considered when the original trial was designed, these results need to be confirmed. Confirmation might include some or all of the following procedures.
- Review results with participants. Participants with their managerial expertise often are able to verity or reject initial hypotheses,
- Evaluate the scientific logic of the hypothesis (i.e., possibly involving outside experts).
- Re-do the trial designed in a manner that the preliminary hypothesis can be tested.
10.4.3 Modified Stability Analysis
In FSD, modified stability analysis (MSA) or adaptability analysis, as suggested recently by Hildebrand and Russell [1994], is being used increasingly to assess the biological performance of treatments (i.e., technologies) in different technical environments. When used judiciously, this analysis also can help assess technical feasibility under different farmers' management systems.
Stability analysis needs data from a multi-environment test. The same set of treatments is tested in each environment. Treatments could be varieties, tillage systems, and so on. Because the testing required for this procedure is expensive, having treatments without proven potential in at least one environment is not cost effective. A range of environments is obtained by conducting the test at different locations and seasons within a country or target area. A range of environments also can be created within one site by imposing another performance factor. For example, if several irrigation regimes are implemented at one site, the set of treatments can be tested in each regime. The test needs to cover the entire range of environments envisioned by the researcher as targets for these treatments.
In on-farm research, multi-environment tests are common, The testing of a technology to improve grain yield on a range of farms, in different villages, and over several seasons gives excellent data for stability analysis.
Stability is measured from a simple linear ( i.e., straight line) regression of the performance for one treatment on the index values of the environments. The environmental index is a value that is supposed to define the standard for performance in each environment. To visualize the analysis, consider a scatter diagram, The indices for environments are put on the horizontal axis and the measure of performance on the vertical axis, The environmental index points are not spaced equally. Two environments might have nearly the same index. The straight line regression shows how much change in performance of the treatment takes place for each change of the environmental index unit,
Performance of crop production treatments usually is measured as grain yield, But, performance also could be measured by other factors, such as crop establishment, plant height, weed levels following tillage treatments, etc. Criteria important to the farmer also could be used (e.g., net return per hectare, etc.), Therefore, in a sense, modified stability analysis potentially could be used to assess more than technical performance,
The regression analysis is easy; obtaining good measures of environmental indices is not, Measurements of rainfall, soil depth, and soil pH are examples of indices that have been used in stability analyses. As an example, stability analysis could show how much the yield of a tillage system changes with each 25 millimetre increase in seasonal rainfall. More often, indices are calculated from the performance of treatments themselves. Suppose yields for double-plough planting are regressed on yields of single-plough planting in data covering 120 farmer-implemented comparisons of these systems. The 120 single-ploughing yields become 120 environmental indices. The regression can explain how much change in double-ploughing yield takes place with each 10 kilogram increase in single-ploughing yield. In this ploughing example, yield of single-ploughing is a good measure of environment when this system is the standard for that area.
Usually, in practice, the standard performance is measured as the average of several or all the treatments. Statistically, this is a problem, because Y. the dependent variable in regression, is part of X, the independent variable. Their values are automatically related. Nonetheless, the average of treatments often is used as a measure of environmental index. Take a test of four cowpea varieties conducted on three farms in each of three villages with the test repeated for three seasons. For each of the 27 environments, the index can be calculated as the average yield of the four varieties. Stability is assessed separately for each variety by regressing variety yield on average yield.
For stability analysis to be of use, the range of environments must be adequate. Even though regression can give a prediction of outcomes for environmental indices beyond those in the data, the prediction is not likely to be accurate for environments that are far outside those of the test. Data drawn only from testing in wet seasons with high yields cannot be used to assess stability over dry and wet seasons. Data for stability analysis need to include a minimum of one environment from each end of a typical range of the indices.
In stability analysis, three regression statistics are used as indicators of stability: the coefficient of regression ('b'), the standard deviation of this coefficient (sb), and the treatment mean.
Because 'b' indicates the slope of the regression, it explains whether the performance of a treatment improves faster, slower, or the same as the environmental index, If the environmental index is expressed in the same units as treatments (e.g., yield), 'b' equals I when treatment and index increase at the same rate. The sb indicates how much the researcher can rely on the regression relationship. A high sb means that the performance of the treatment is erratic in good and poor environments. The mean, combined over environments, indicates the general ability of a treatment to perform well.
The stability or adaptation of treatments (i.e., technology) is defined as follows:
- With good stability or general adaptation, 'b' equals 1, mean performance is high, and sb is low. Other researchers feel that stability is highest when 'b' is less than 1 (i.e., the performance does not change much between poor and good environments), mean performance is high, and sb is low,
- With good specific adaptation, 'b' is significantly different from 1 (e.g., less if adapted to poor environments and more for better environments). To be considered good specific adaptation, the performance must be relatively high in the environments for which the treatment is adapted and the sb low.
- Poor adaptation is indicated by low performance means, regardless of the regression coefficient ('b') or sb.
- Erratic performance is indicated by a high ski, regardless of performance mean or regression 'b'. Note that the performance of a treatment sometimes appears erratic because the treatment is interacting with factors that are not parts of the environmental index (e.g., a new sorghum hybrid does poorly in an otherwise good environment because of an adverse reaction to a soil factor).
The steps in adaptability analysis are the following:
- Conduct a trial with a common set of treatments in each environment, Usually in FSD, each environment is a different farm. It is strongly advised that the set of treatments include the farmer's conventional practice as a basis for making comparisons with farm environments outside of the trial.
- Regress treatment yields on the environmental index of each farm.
- Differentiate treatments that are favoured in the high-yielding environments from those that are superior in the medium- and in the low-yielding environments. It should be noted that these differences, which emerge as differences in regression slopes, may not be real if statistical tests for difference are not significant.
- Characterize high- and low-yielding environments further by looking for a non-experimental factor (e.g., soil factor, climatic factor, etc.) that is linked to the responses. The practical expertise of FSD staff and farmers is needed to identify factors that logically should link to different responses of trial treatments. To evaluate the factor-response relationship in an observational manner, the factor values can be plotted on the same regression graph used in the adaptability analysis. It is advisable to discuss with a statistician other tests (i.e., analysis of covariance) that might be used to assess how strong the relationship is between a factor of the environment and the trial responses. The factor that is chosen then is used to define the appropriate recommendation domains for the technologies that were tested in the trial.
This method of sorting farm environments works quite efficiently, if the causes of high or low performance are constant from year to year. However, in some environments, variation in weather or some other factors exerts a large influence on the relative performance of many treatments, This means that the relative performances of treatments might place the farm in one recommendation domain in one year and in another domain in the next season, Therefore, the use of adaptability analysis would need more years of data in areas with highly variable weather than in growing environments with more equable weather.
BOX 10.1: MSA CAN HELP DEFINE RECOMMENDATION DOMAINS
Farmers in the Manaus District of Brazil clear forested land to plant crops such as cowpeas. Soil fertility declines quickly and, after a few seasons, the land is abandoned. Research was conducted to define appropriate soil fertilization recommendations for this zone and different circumstances in this zone [Hildebrand and Russell, 1994]. MSA was used for data collected on 13 farms that each tested four potential fertilizer packages. This included one package that was the farmer's own practice of not applying fertilizer. For these farmers, cash is scarce and the major constraint to development, so the focus when assessing response was on the criterion of kilogrammes of cowpea per dollar of cash cost.
Following a step-by-step application of MSA to this problem, two recommendation domains were delineated and factors within the environments that characterize these two domains were identified. The message for extension of this MSA is that:
Because of its high cost, a full dose of triple superphosphate fertilizer would be too risky for most farmers. If it is used, it should be used only for the first or second seasons of planting.
10.4.4 Computer-Based Simulation
As of today, the use of simulation tools is not common in FSD technical analyses. However, the use of such knowledge-engineering methods has proved helpful in evaluating alternatives in other fields. These simulations permit the users to evaluate multitudinous alternatives, at least on a preliminary basis, without investing in the implementation of real-life trials. Simulation scenarios could involve technology or management alternatives, changes in the future circumstances of farming, and so forth. Simulation programmes in the future might prove to be a very effective means of organizing FSD data. These programmes might strengthen the linkage between knowledge systems (e.g., allowing participants to effectively interface with scientific knowledge frameworks).
Simulation is a developing area, however, and good tools are not presently available for most FSD analytical work. Criticisms of contemporary simulation models for use in FSD focus on their inability to:
Solutions to some of these problems likely will be found. It is anticipated that simulation programmes, probably supporting an iteration between machine and human participants, will become part of the arsenal of FSD analytical tools of the future, particularly with respect to FSD with a 'natural resource systems focus' and with a 'livelihood systems focus' (Section 3.3).