There is a growing body of monitoring and evaluation (M and E) literature particularly relating to agricultural development projects and programmes. However this provides little guidance on what information is needed to monitor and evaluate changes in the bio-physical conditions of a project area. To date M and E within an agricultural development context has generally focused on assessing project performance measured in terms of productivity and short term socio-economic benefits, rather than assessing long-term bio-physical sustainability. As a result experience with the use of bio-physical information in an M and E capacity has been limited. Similarly there is limited experience with the development of M and E systems for monitoring and evaluating the effectiveness of SARM related projects (see box 52).
Monitoring and evaluation, in the context of SARM projects and programmes, are interrelated processes concerned with the collection and analysis of information of ongoing and completed activities. They are an essential part of SARM as by documenting and analysing the results of any interventions, they enable an assessment of whether they are successful in tackling identified agricultural production and sustainability problems. If they are not, or not as successful as anticipated, then the M and E exercise should help identify why.
In some cases the assumptions underlying recommended SARM technologies may have been inappropriate, household circumstances may have changed or been misunderstood, the potentials of particular enterprises and management practices within the farming system been misjudged. By feeding the information and results of the analysis back for further analysis and development work it is possible to refine or change the recommendations and thereby improve their chances of being adopted and achieving the hoped for benefits.
Monitoring for SARM involves the systematic collection of data to provide information on the implementation of any research and dissemination activities (implementation monitoring) and as the basis for ongoing and ex-post evaluation (impact monitoring). Implementation monitoring provides management with information on the utilisation of research and extension inputs, timing of activities, and timely generation of outputs. Impact monitoring is more directed to effects, positive or negative, on both the socio-economic circumstances of the participating households and their bio-physical resources (soils, vegetation, water and others).
Impact monitoring is concerned with collecting data to detect and follow change within an area of concern e.g. a village or watershed. Change may result from adoption of the recommendations of a SARM project, or due to other factors. Impact monitoring will include collecting environmental data such as rainfall (total and distribution) and measuring (where equipment is available), or qualitatively estimating by direct observation, changes in soil properties including top soil and nutrient losses, changes in vegetation such as ground cover and species composition, and changes in the quantity and quality of surface and groundwater resources. Impact monitoring may also involve conducting regular surveys to collect data on crop, livestock and tree performance and to record details of rural household land use and management practices. Questions to be answered by impact monitoring should include:
Evaluation is undertaken using the data obtained from the implementation and impact monitoring exercises. This data is compared with both the development and immediate objectives of a SARM project or programme, as well as with the work plans and expected results (targets) of the research trials and the dissemination activities undertaken, to see if they are being achieved.
An evaluation into how successfully the planned research and extension activities were implemented may be positive while an evaluation of the impact on rural households may be negative. In other words the evaluation may reveal that although the project staff organised their work programme in an optimal manner, the inputs and proposed innovations did not result, as expected, in positive effects on the rural household production and sustainability problems. The basic tasks in evaluation should include:
The implementation of a SARM project or programme will be expected to produce on-site benefits for rural households engaged in small-scale farming and forestry activities within the target area. The household members will be the best informants as to whether the expected results are occurring in the field. Hence there is a need for what is referred to, in M and E circles, as beneficiary contact monitoring. This will make it possible to gauge whether the extension recommendations are relevant at the household level and are actually followed as intended. Beneficiary contact monitoring includes items relating to issues such as awareness of the recommendations by different rural household groups (common resource management domains), actual adoption of technical recommendations, reasons for non-adoption, degree of adopter satisfaction, and so on.
Depending on the nature of the area, various types of key indicators are applicable. For most SARM exercises, the checklist indicators given in Table 13 may be used as a guide in determining the scope and frequency of beneficiary level information to be collected. Such data are generally collected through surveys of rural households, which may be done formally or informally.
While the use of a statistical sample is preferable, for analysis purposes, this may not always be possible given the manpower and financial resources available. Hence, short-cut approaches and the use of rapid rural appraisal techniques may sometimes be necessary, particularly as a fairly large margin of error can often be tolerated.
Those undertaking the M and E work should not lose sight of the fact that the main purpose of the information is to help alert them to any potential problems from the beneficiaries' perspective, so that corrective action may be instituted. Unless data from a survey could be collected and analyzed quickly, then they are as good as useless, and it would be far more effective to rely on other methods of information gathering than to attempt a farm level survey.
The indicators used should ultimately be determined by the nature of the objectives and intended impact of the recommendations. The objectives need to be clearly defined and, as appropriate according to whether they are realisable in the short-term, intermediate or long-term. Short-term objectives could relate to increasing yields of annual crops, intermediate to the production of firewood and improving productivity of natural pastures and the long-term to reducing rural poverty, improving health through better nutrition, preventing land degradation while ensuring sustainable use of the natural resources.
Table10. Beneficiary contact monitoring: list of some possible objective indicators and their suggested recording frequencies*
Type of Indicator |
Examples |
Suggested Frequency |
1. Adoption of conservation effective farming practices |
a. Land clearing practices b. Tillage methods and practices c. Planting and Weeding practices d. Crop rotation practices |
Yearly |
2. Changes in land use, cropping patterns and enterprise production levels |
a. Crop areas cultivated per farm b. Cropping intensity per farm c. Crop type and sequence d. Crop yields and marketed production e. Livestock nos. per unit area f. Livestock outputs and sales |
Repeated seasonally or yearly |
3. Changes in sources of timber/fuelwood |
a. Collection /harvesting /purchase from farmers own holdings or community woodlots b. Collection from traditional sources (areas at risk) |
After a period of 3 years and then yearly |
4. Adoption of on-farm tree planting and agroforestry soil stabilising techniques |
a. Forestry seedlings obtained and planted b. Silvi-pastoral species obtained and planted c. Extent of planting in contour hedgerows, on bunds and terrace risers d. Survival rates of plantings and replacement of failures |
Yearly |
5. Changes in sources of animal fodder |
a. Establishment of cut and carry plots b. Utilisation of fodder from surrounding areas |
Every 1-2 years |
6. Changes in patterns of transhumance |
a. Location of wet and dry season pastures b. Herd sizes and movement (in relation to areas at risk) |
Every 1-2 years wet and dry season |
7. Maintenance of physical structures |
a. Status and condition of bunds/terraces /storm drains /waterways b. Incidence and frequency of repair operations c. Method used for repairs and maintenance |
From around the second year after construction then every other year |
8. Use of inputs and support services |
a. Purchase of improved seed for crops b. Hiring of tractor/oxen services c. Sought advice from extension service) |
Seasonally or yearly |
9. Participation in community activities |
a. Voluntary community level construction, and maintenance of conservation works, revegetation and bush clearing of grazing areas b. Voluntary restraint on livestock numbers and movement |
Periodically as required |
10 Changes in Family Welfare |
a. Household food self sufficiency b. Health status c. Income levels d. Standard of housing |
Every 1-2 years |
* All information to be obtained at the beneficiary (rural household) level.
Source Douglas and Lai 1988
There is a need to be able to monitor and evaluate changes in the bio-physical properties of the land within a project area in order to determine whether or not they are beneficial and whether they have come about as a result of the project's activities. It is rarely possible for routine SARM project monitoring purposes to install sophisticated measurement techniques to detect changes in the bio-physical environment, e.g. to obtain quantitative figures for changes in the amount of run-off or soil loss during the life of a project. This can be done for small catchments under controlled research conditions (see Lawrence and Dickinson 1995 for examples from the Asia region) but not for the relatively large areas that would be covered by SARM extension projects.
It is essential that the M and E system should capture the bio-physical dimension of any SARM project. Such bio-physical data is needed for two major purposes. Firstly to determine the extent to which project performance is influenced by the bio-physical factors at play within the different parts of the project (e.g. seasonal and spatial variations in climate, as well as differences in slope and soil type will all affect the impact of specific project interventions). Secondly to monitor and evaluate changes in some of those factors (e.g. soil productivity, soil erosion) as a result of the project's activities. A knowledge of the variation in bio-physical conditions within a project's area is essential for M and E purposes in order to determine the extent to which project performance is influenced by the beneficiaries natural, as opposed to socio-economic, circumstances. A failure to achieve target outputs may not be because project staff failed to organise their work programme in an optimal manner. Instead it may be that specific technical recommendations (e.g. hedgerows and rockwalls on steep slopes) were unsuited to the prevailing bio-physical conditions of particular project localities.
It is believed that the impact of a SARM project on the bio-physical environment should be monitored on at least the following three levels:
There may be instances when larger areas need to be covered for instance in China, following the passing of the 1991 Soil Conservation Law, the agency responsible for soil conservation within each Province is required to provide updates on the status of soil erosion in its province to the central government every three years (Douglas et al 1996). Also required is information on soil and water conservation enforcement issues and actions taken within the province.
Essential for any bio-physical impact monitoring would be the compilation of baseline data sets on the bio-physical conditions within each of the localities where a SARM project has field activities. Such bio-physical databases can often be compiled from existing secondary data sources and the knowledge of key local informants, however they may also require some basic natural resource survey field work. It is surprising how many projects claiming to be promoting SARM have very limited data on the bio-physical conditions present, which hampers the assessment of the true bio-physical impact. Baseline bio-physical data sets can usefully be used for defining, and demarcating onto topographic base maps, the boundaries of the individual land management units (LMUs) to be found within a project area. This would provide a basis for stratifying environmental impact and adoption rates of the different project interventions.
The boundaries of the geographic areas used for macro level monitoring should, when ever possible, conform to the social and cultural boundaries of the communities with which the project is working. To many soil conservation projects still use topographic boundaries (the micro-/sub-watershed) which is missing the reality that many of the participating farmers may have fields outside the area being monitored.
The overall conservation effectiveness of farmers land use/management practices (see chapter 9) should be used as the basic criteria for determining the environmental impact of the project interventions. Monitoring should not just focus on the impact of individual interventions but consider how they interact within the farm household system. For instance concern with better land husbandry means that how the land is used between hedgerows and rock walls, or in the catchment area upstream of a plugged gully, needs to be assessed to determine the overall environmental impact and conservation effectiveness of field level improvements.
Crop yields, which are usually monitored as part of the beneficiary contact monitoring component of a project, can be used as a proxy indicator of soil productivity. Such yield data would need to be plotted over time (minimum 5 years) to determine the mid to long term impact of a SARM project's interventions on soil productivity. Simple visual indicators could be used during participatory transects to assess changes (long term trends) with regard to the status, type and severity of soil erosion.
Several soil conservation services already have simple keys for visual scoring of the current state and severity of erosion, range ecologists have similar techniques for assessing the quality of rangelands (species composition and percentage ground cover), and many agronomy departments have photographic keys for detecting plant nutrient deficiencies (possible proxy indicator of soil nutrient problems). Techniques such as these could provide the basis for simple qualitative methods that could be used by field technicians and farmers for monitoring long term changes (trends) in the quality of the land resource as part of a participatory approach to project planning, monitoring and evaluation.
Monitoring the impact of project interventions only on an individual farm plot basis will fail to capture the full environmental impact on the overall management of natural resources at the farm household level. This is because it cannot be assumed that the impact of individual interventions, as measured on one plot, will universally apply to the whole farm holding. Thus the environmental impact at the farm household level could be monitored through the compilation of an annual (or as appropriate seasonal) land husbandry status report/farm inventory for a representative sample of the participating farm households.
In addition to monitoring at the micro/individual plot and farm household levels there is a need to monitor the environmental impact on a geographic area basis. This to allow for the determination of the areal extent of any changes (positive or negative) in the nature and severity of existing land degradation within a project area. One option would be to compile for each geographic area monitored an annual land husbandry status report/inventory that would document both quantitative and qualitative environmental changes (Douglas 1995c). Much of the data for this could be collected on a direct observation basis by M and E staff undertaking representative participatory mapping and transect exercises with farmers and field agents within the monitored area.
Given the practical difficulties in obtaining quantitative bio-physical data it is believed that greater use should be made in M and E of qualitative data. The idea being to detect trends i.e. to reveal whether, following implementation of the project's activities, the processes of land degradation are continuing, if so then is the underlying trend, from a soil conservation/productivity perspective, one of a worsening or improving situation? In this regard it is believed that there is a need to find simple visual parameters, or proxy indicators, to obtain a qualitative estimate of the severity of existing land degradation, for a specific locality and at a particular moment in time. By repeating such observations over a period of time (e.g. every 1-2 years or at 2-3 week intervals during a cropping season) it should be possible to assess the underlying trend and to determine whether, following the adoption of the improved land use practices promoted by the project, land degradation has actually been reduced.
There are a growing number of papers and manuals describing the use of simple qualitative techniques that could be used for assessing the extent and severity of land degradation. For instance a variety of simple reconnaissance methods for estimating soil loss are described in chapter 2 of the FAO Soils Bulletin No 68 Field Measurement of Soil Erosion and Runoff. Likewise the Centre for Development and Environment at the University of Berne in Switzerland has recently published a manual for the field assessment of current erosion damage (Herweg 1996). Guidelines also exist for the qualitative monitoring and evaluation of better land husbandry at the field level and for the compilation of area status reports (see Douglas 1995 and FAO 1996).
Project documents typically record adoption rates for such interventions as contour hedgerows, rock walls, and gully plugs, on the basis of the number and length of each that have been installed at the field level. It is rare to find that any attempt has been made to monitor the quality of the initial establishment/construction or whether there has been any subsequent maintenance.
Project reports will often record that they have been able to conserve so many hectares, based on the size of the area in which conservation measures have been installed. However such assessments are typically made on the assumption that all of the reported measures are functioning as designed. Experience from field visits to a number of such projects suggests that the reality on the ground may be degraded hedgerows, poorly maintained rock walls and gully plugs, and/or eroding terrace risers.
Hence monitoring needs to be able to rate the quality of the various conservation measures installed during the life of a project in order to arrive at a more accurate overall assessment. One option is to develop a quality rating index, to be used to determine the conservation effectiveness of the project interventions. This was done in the USAID/World Bank funded Upland Agriculture and Conservation Project in Central Java Indonesia for monitoring terrace quality (see Gnagey 1991).
The above discussion on M and E has focused largely on the conventional approach in which the socio-economic and bio-physical effect on the beneficiaries of a project is monitored and evaluated by `outsiders' (Government workers, project staff etc). However whenever possible the beneficiaries should be directly involved in this work. This would require the adoption of a participatory monitoring and evaluation approach.2
Participatory monitoring would involve groups of farm households collecting, processing and sharing data, amongst themselves concerning the implementation of SARM recommendations (e.g. on farm better land husbandry practices) and the condition of the land resources within their village/community area. The role of the project research and extension workers would be to facilitate the data collection, by providing technical advice and training, as well as being in a position to promote the exchange of information between different groups of land users and as appropriate project management.
The main tools for participatory monitoring could be:
By using symbols rather than words, on farm and group level forms and charts, even illiterate farmers can participate in collecting and recording their own data. Participatory monitoring is a process through which the beneficiaries of the field level innovations of the catchment management plans can educate themselves, and others, while retaining control over the data collection and analysis. They thus become active rather than passive participants with a real interest in the outcome.
Participatory evaluation is the systematic analysis, of the monitored information, by the beneficiaries with the assistance of the project staff with a view to:
Participatory evaluation is expected to take the form of discussion sessions during participatory village level workshops while consolidating the farm level monitoring data. It would also take place during participatory transects while discussing observed changes in farming practices and the nature and severity of any visible land degradation. Evaluation done this way stimulates critical self-awareness amongst the farm households of the impact of their farming practices and can provide the motivation to make improvements.
25 See FAO 1988 and 1989, FAO RAPA 1988 for a discussion of the concepts and procedures for participatory M and E and details of a range of alternative tools and techniques that can be used.