Climate Smart Agriculture Sourcebook

CSA programme and project monitoring and evaluation

Enabling Frameworks

Challenges and principles in monitoring and evaluation

There are many challenges and principles that need to be considered in assessment, monitoring and evaluation for agricultural development projects and programmes. This module does not discuss these general issues. Interested readers are referred to IFAD, 2002; World Bank, 2005; World Bank, 2006 and FAO, 2010. Climate-smart agriculture poses unique challenges for assessments, monitoring and evaluation. The following sections, which sets out the guiding principles for meeting these challenges, are intended to highlight specific problems that are particular to climate-smart agriculture. No concrete approaches for assessment, monitoring and evaluation are prescribed in this module, as each climate-smart agriculture project and programme is context-specific. Instead climate-smart agriculture programme or project designers are encouraged to be aware of the challenges and to follow the principles laid out in this section. Most of the challenges and principles are common to assessments for policy and project design, as well as monitoring and evaluation.

C9 - 3.1 Definitions and goals of in monitoring and evaluation

Climate-smart agriculture means different things to different countries, depending, in large part, on the level of agricultural development. In some cases, more focus is placed on mitigation, while in others the focus is mainly on increasing productivity and enhancing resilience. One of the first steps for successful climate-smart agriculture interventions and their monitoring and evaluation activities is to define the broad climate-smart agriculture goals with the wide participation of different stakeholders, and then agree on the specific elements of the intervention. Some assessment systems, for example the World Bank's climate-smart agriculture indices (Box C9.4) and the FAO Sustainability Assessment of Food and Agriculture systems (SAFA), attempt to capture the complexity and multiple aims of climate-smart agriculture. 

C9 - 3.2 Situating monitoring and evaluation within a broader development perspective

To avoid duplication, monitoring and evaluation systems should be built on and integrated into existing systems, programmes and projects for agriculture, climate-responsible development and disaster risk reduction (Hedger et al., 2008; GIZ, 2011a). Within agricultural and rural development projects, there are already many actions, expected results and indicators that incorporate information on climate change actions and outcomes, or that can be enhanced by climate-smart agriculture actions with relatively lower costs (see FAO, 2012a). For guidance on participatory approaches, see the FAO Socio-economic and Gender Analysis Field Handbook (FAO, 2001). The introductory section and sections on how to do monitoring and evaluation refer to several sources (e.g. FAO Investment Learning Platform) for guiding broader monitoring and evaluation for agriculture and rural development programmes and projects. 

C9 - 3.3 Scales, leakage, permanency, externality and ancillary impact

Climate change interventions implicitly address longer-term and larger-scale processes. They also involve a greater number of potential trade-offs. For example, additional irrigation represents a valuable adaptation method to overcome longer droughts, but higher efficiency requires the additional use of energy, which can increase greenhouse gas emissions. Most efficient irrigation systems (e.g. drip irrigation, micro-irrigation sprinklers) require equipment that is currently powered by fossil fuels. Unlike many projects where monitoring and evaluation addresses areas, beneficiaries and stakeholders within the project’s ‘boundaries’ for a relatively short period after the project ends, climate-smart agriculture projects are more likely to require longer-term post-project monitoring of trends and comparison areas. As climate change initiatives cannot be developed or implemented in isolation, multicriteria and multiple objective analyses can help to assess trade-offs, and guide the subsequent monitoring and evaluation of chosen interventions.

Some expected outcomes and impacts may not be able to be evaluated during the course of the project or immediately after. Some assumptions on longer-term benefits may need to be incorporated in the evaluations. This is particularly true for the monitoring and evaluation of mitigation benefits. Increases in soil carbon content as a result of climate-smart agricultural practices will not continue indefinitely. Eventually, soil carbon storage will approach a new equilibrium at which point carbon gains equal carbon losses. A default time period, usually 20 years, is assumed for this transition.

Similarly, the issue of leakages and permanency is important for the monitoring and evaluation of climate change mitigation. Permanency refers to the principle that emission reductions represented by an offset should be maintained over time. In some cases, abandoning a climate-smart agriculture practice after only a few years will counterbalance the emissions previously avoided, and sometimes it may even surpass the emissions abated. This is why frequent monitoring is required to take into account such risks. Leakage refers to a situation where the emissions abatement that has achieved in one location is offset by increased emissions in unregulated locations. In this regard, the difficulty lies in the choice of appropriate boundaries to conduct the appraisal.

A measure adopted for climate-smart agriculture may bring short-term benefits, while the same measure may lead to maladaptation over the long term and vice versa (Hedger et al., 2008; Villanueva, 2010). The timing of monitoring and evaluation needs to be chosen to address both short- and long-term impacts. Different targets may be set for different time scales. Considering pathways for implementing climate-smart agriculture at different time scales will help improve the design of monitoring and evaluation systems. Ideally, additional evaluations are done after the project ends. Institutions should have adequate systems for storing and retrieving information to support monitoring and evaluation (Lamhauge et al., 2011; Hedger et al., 2008).

Accounting for externalities and ancillary impacts should also be considered, even if they are far more difficult to evaluate than the abatement of greenhouse gases or improvements in adaptive capacities. Virtually every climate-smart agriculture option will produce some positive impact (e.g. clean water or more pollinators) or negative externality and/or ancillary impact (e.g pollution or loss of biodiversity). Whether quantifiable or not, these impacts represent real costs or benefits and should be factored into the monitoring and evaluation process.

C9 - 3.4 Attribution of results 

The attribution of impacts (e.g. adoption of technologies) can be difficult to evaluate with most monitoring and evaluation systems. This has implications for the way project impact evaluations are designed and the tools that are used. Factoring in the effects of climate change makes this issue even more challenging.

Climate is variable by nature. The weather experienced daily is a combined result of natural climate variability and anthropogenic climate change. It is difficult to separate the two for the purposes of assessing the impacts of climate change or monitoring and evaluating the impacts of climate-smart agriculture interventions (Lamhauge et al., 2011; Hedger et al., 2008). It is also not easy to clearly distinguish the effects of many adaptation options from those achieved by broader sectoral development policies (UNFCCC, 2010; Lamhauge et al., 2011). The distinction is especially unclear when climate change adaptation interventions are not designed and implemented as stand-alone projects or components, but incorporated into various development activities. Indicators for the successful implementation of climate-smart agriculture that can be attributed to a specific intervention should ideally reflect achievements in addressing the additional impacts of climate change, such as the capacity to cope with increased frequency and intensity of natural disasters over the long term.

It should also be noted that climatic risks are not static. The baseline situation and baseline projections against which impacts of climate-smart agriculture are evaluated may change as climatic conditions change (Hedger et al., 2008; Lamhauge et al., 2011). Frequent updating of a ‘moving’ baseline with new information on climate, hazards, extreme events, and their impacts on agriculture is necessary to make the appropriate adjustments to climate-smart agriculture interventions and their targets (Lamhauge et al., 2011; Hedger et al., 2008; Villanueva, 2010).

C9 - 3.5 Challenges in gathering a comprehensive range and long term data and information for climate-smart agriculture

For monitoring and evaluation, data need to be collected throughout the climate-smart agriculture intervention and after all its activities have been completed. However, data collection is difficult and costly, particularly for smallholder farmers (Lamhauge et al., 2011; UNFCCC, 2010) and many local institutions. Monitoring and evaluation is already a challenging undertaking for regular development projects. It is important to address data overload (i.e. too much information with too little useful analysis) by simplifying monitoring and evaluation processes and indicator sets wherever possible (see also GDPRD et al., 2008) and maximizing the use of existing systems. Box C9.10 provides a case study on how focusing on measuring land-use change within a project can simplify the monitoring of carbon sequestration and adaptation.   

The key point is to identify the most relevant indicators (see examples of indicators in Table C9.2) for project monitoring and evaluation purposes and broader policies and programmes (Figure C9.1) and to continue to collect data for these indicators. These indicators will have to balance minimum information requirements with some standardization for comparability. Some of the benefits of climate-smart agriculture interventions may not be realized for a long time – perhaps for decades – much longer than the timelines typically associated with projects. Supporting the collection of associated data for the purposes of evaluation beyond the project is a serious issue (Hedger et al., 2008; GIZ, 2011a). Commitments to set aside resources for this should be considered as a means of providing a global public good.

In many developing countries, improving information and data collection and availability is a priority. Targeted climate-smart agriculture strategies and interventions need to be based on reliable user-oriented information that includes good quality data, documented vulnerabilities and accurate evidence. Emerging information technologies can provide new opportunities for more efficient and accurate data collection (see Box C9.9). 

Box C9.9  Role of information and communications technologies, and communication for development

Information and communications technologies are important for implementing climate-smart agriculture, particularly for monitoring and evaluation. These technologies are central for the collection, processing and transmission of data. They also allow stakeholders to communicate easily among themselves. GPS equipment used in project officers’ cameras can automatically log the locations of the photos taken for later reference. GIS is essential in analysing geo-referenced information. Collected information can be logged in the database for monitoring purposes using simple structured forms based on a mark-up language (e.g. XML) on mobile phones, mobile electronic devices and laptops.

The Mobile Survey Tool, developed for the Ericsson Millennium Villages Project, is an example of a tool that facilitates data collection for agriculture, health care, business, finance and government. It enables operators and end users to create and organize surveys and questionnaires without the need of coding or databases. The data can then be processed and used for different purposes within a village or by governments.

C9 - 3.6 Adapting system and enhancing Capacity for assessment and monitoring and evaluation

Inadequate capacities (technical, institutional) and resources (human and financial) are often cited as barriers to successful assessment, monitoring and evaluation activities (UNFCCC, 2010). The trend has been to use country-driven systems. To make these national systems effective there is a need to strengthen individual and institutional capacity through effective capacity development, such as individual training in the area of data collection, assessments, monitoring and evaluation for climate-smart agriculture (see module C1 on system-wide capacity development).

Monitoring and evaluation often have considerable transaction costs. Unless appreciated as a useful tool by stakeholders, monitoring and evaluation can be seen as a burden that offers little value for the effort involved in gathering significant amounts of information.

There are limited choices for appropriate analytical methods for assessment, monitoring and evaluation that address the specific needs and conditions of climate-smart agriculture projects. However, there is a considerable body of experience from natural resources management and rural development projects that monitoring and evaluation activities in climate-smart agriculture programmes and projects can build on. Many of the existing tools and models are intended for highly skilled technical experts in academic institutions. They may not be suited for implementing climate-smart agriculture in developing countries. Further collaboration and communication between the developers of the tools and their users are necessary to ensure that simple tools that meet the needs of climate-smart agriculture practitioners are available. 

Some tools may be less sophisticated and produce less detailed scientific results but still meet the needs of the climate-smart agriculture community. It is necessary to find the right balance between the simplicity of the tools and the reliability of the results.