Climate Smart Agriculture Sourcebook

CSA programme and project monitoring and evaluation

Enabling Frameworks

How to design and implement monitoring and evaluation for climate-smart agriculture programmes and projects

C9 - 2.1 Overview of the planning, monitoring, evaluation and learning cycle within a climate-smart agriculture programme 

As indicated in chapter C9.3 the monitoring and evaluation framework and systems are designed once the assessments of climate change scenarios have been made, or are at least when the climate-smart agriculture intervention options and detailed project or programme plans are being formulated (see also module C8 on climate impact assessments and appraisals of climate-smart agriculture options and module C1 on system-wide capacity development). The monitoring and evaluation cycle consists of key elements that are linked into the whole programme and project cycle.

  • Monitoring and evaluation of climate-smart agriculture programmes and projects use as a starting point the baseline projections regarding climatic conditions, even if these projections are preliminary. They are also based on the desired climate-smart agriculture objectives stated in the policy and project design assessments, which include an assessment of system-wide capacities. 
  • At the same time, given the uncertainties of climate change and the constant need to adapt to these uncertainties, as well as other factors of complexity, a more adaptable programme process may be needed. This will include developmental evaluation, where strategies and indicators may need to change on an ongoing basis through project implementation. This must be done in a highly participatory way to foster country-ownership and commitment for mutual accountability of results. 
  • Monitoring and evaluation are initiated at the project preparation stage, as indicated in Figure C9.1, when there is an interplay between assessments, monitoring and evaluation activities. These activities are intimately linked through detailed and regular planning processes. In particular, impact evaluation frameworks should also guide the preparation of project and programme baselines. 
  • It should also be noted that there may be different levels at which data for climate impact assessments are gathered (e.g. national, regional, landscape and local), and for which interventions are designed. The predicted climate impacts at each level may differ somewhat, with those at the finer-grain levels being more specific and even unique. Objectives, indicators and baselines at the national or programme level may be quite general. However, at a more local level, they will need to be increasingly tailored to the context and the specific nature of the project intervention. A hierarchy of objectives and indicators might then be developed with monitoring and evaluation data from a range of different unique local projects being combined to prepare a report on indicators at the programme level. 
  • Shortly after appraisal of the project proposal and approval of the project, detailed indicators, baselines and targets are set, with clearly specified beneficiaries and well-defined interventions. Commonly used indicators covering a range of important climate-smart agriculture aspects are given in Table C9. 3. The identification and selection of indicators are further discussed in Box C9.4.
  • Throughout the implementation of the project, the progress of climate-smart agriculture interventions against indicators is monitored, as is the use of resources and delivery of outputs. 
  • At the mid-cycle and end of the project, the impacts of climate-smart agriculture interventions on socio-economic, environmental and livelihood indicators are evaluated based on the baseline situation and the initial expectations in terms of results. 
  • As some of the benefits of climate-smart agriculture may not be realized within the timeframe of a short project, but only during a subsequent capitalization phase, it is ideal to continue project monitoring and evaluation and adaptive management beyond the project cycle, and institutionalize it in ongoing programmes.

The FAO Investment Learning Platform provides concise and practical guidance for planning, formulation, implement and evaluating public investment in agriculture and rural development. A number of manuals and e-learning tools are available for in-depth monitoring and evaluation that can be applied to climate-smart agriculture interventions. Examples include the International Fund for Agricultural Development (IFAD) Monitoring and Evaluation Guide (IFAD, 2002); the World Bank’s monitoring and evaluation tools and approaches, with basic definitions provided by the Organisation for Economic Cooperation and Development, Development Assistance Committee (OECD/DAC, 2009); and the European Commission Project Cycle Management. 

Any monitoring and evaluation system needs to be developed using the basic process laid out in the following paragraphs. The process is elaborated in FAO 2012b and draws on work carried out by GIZ (2011a), which specifically looks at monitoring and evaluation of climate change adaptation interventions. Much of the emphasis is on the effective preparation of monitoring and evaluation activities and the development of a strong framework and adaptive capacity to measure progress and change. 

  1. Conceptualization. Situation analysis will build upon climate impact assessments and climate-smart agriculture options assessments (see Figure C9.1 and module C8 on climate impact assessments and appraisals of climate-smart agriculture options) and broader assessment baselines, together with an initial review of resources, key institutions and implementation mechanisms that form the concept for a detailed intervention, usually for a project or programme (see also module C1 system-wide capacity development);
  2. Preparation and appraisal. Programme and project intervention planning and targeting sets the detailed framework within wider programme and project cycle management. Project cycle management encompasses a broader framework of strategic planning, detailed project planning, implementation, monitoring, evaluation, learning and re-planning, and influences existing and new programmes. Detailed planning activities that are important to monitoring and evaluation include:
    1. Identifying the contribution to adaptation and/or mitigation: this helps determine specific areas for engagement, such as adaptive capacity, adaptation and/or mitigation actions, and sustainable development in a changing climate.
    2. Forming an adaptation hypothesis and theory of change: this is required to delineate, in a participatory, gender-responsive way, the possible options and their expected changes, and the results chains between activities, expected behaviour changes, outcome and impacts. These can then be formalized in the intervention design and process, often in the form of logical frameworks that outline indicators, assumptions and risks to achieving these changes. These will help define
      • inputs and activities (the details and resources of the actual interventions);
      • outputs (the direct results and deliverables of the interventions that are required for the outcomes);
      • purpose-level and intermediate outcomes (the expected external changes from the intervention); and
      • higher-level outcomes or impacts that interventions may contribute to, usually affecting household and individual living conditions, and changes in the environment.
    3. Accounting for complexity, 'expecting the unexpected' and adopting a developmental evaluation approach: this involves incorporating an evaluator or evaluative thinking in a long-term and ongoing process of project and programme conceptualization, design, experimentation, adaptation and development, which is sensitive to unintended outcomes, and where evaluative questions, data and self-reflection steer decision-making in the developmental process.
    4. Understanding starting conditions through detailed programme baselines: Detailed data collection of livelihood activities, land use, the household situation of potential beneficiaries provide a rigorous baseline for comparing the impacts of the intervention over time.
    5. Developing a results-based management: this provides a framework whereby monitoring and evaluation is used to encourage stakeholders to focus more on results (outputs and outcomes) rather than inputs and activities. This occurs when potential indicators have been identified.
    6. Developing adaptation and mitigation associated indicators: the indicators are developed in relation to the above hypothesis and changed expectations, and reviewed on a regular basis (see Figure C9.2). Milestones and targets help to identify the range of achievements expected in short- and long-term scales. Project and programme baselines are then prepared to measure future changes.
    7. Carrying out appraisals: these appraisals review the whole design with regard to its risks, technical and social feasibility, robustness and efficiency and safeguards.

A results-based framework (Figure C9.2) indicates how specific climate-smart agriculture practices are expected to be linked to intermediate variables at the output and outcome level, and ultimately lead to improvements in terms of adaptation, mitigation and food security. The starting point is the implementation of specific climate-smart agriculture farming methods or natural resource management practices, as well as other activities that support climate-smart agriculture. An evaluation has to demonstrate if, and to what extent, the activities deliver positive impacts in terms of climate change mitigation, adaptation and food security. Changes in the biophysical, socio-economic and institutional setting may occur as a result of climate-smart agriculture activities, which have been delivered by key project outputs. The outcome variables can usually be defined as changes in behaviour, agricultural systems and institutional capacity that translate into effective adaptation, mitigation and food security benefits. The links between the different interventions, outputs and results and objectives, provide an overall logic and ‘theory of change’.

Figure C9.2.  Linking activities and benefits of climate-smart agriculture through a results-based framework that provides an entry point for the development of indicators

C9 - 2.2 Preparation of monitoring and evaluation of climate-smart agriculture programmes and projects

Monitoring and evaluation start with the design process and the identification of objectives. Before describing monitoring and evaluation indicators and tools, some important design processes need to be outlined. A shared process for setting objectives and determining indicators among intervention stakeholders is key to obtaining feedback, learning and re-strategizing – all of which are important for climate-smart agriculture.

This chapter largely refers to outcomes and impacts because they tend to be specific to climate change and agriculture, and because behavioural, institutional and policy changes, outcomes and impacts are pertinent to evaluation. On the other hand, the project and programme outcomes, outputs and activities to be monitored, which often focus on capacity development, organizational systems change, infrastructure and policy support, will be highly intervention-specific and will fall within more regular planning and monitoring guidance. Also, under changing climatic conditions it is important to also allow for adaptive management, and enhance capacities in this area, both for implementers and the affected communities. This demands that a balance be struck between formal structured frameworks and more flexible approaches. 

Developing the elements for monitoring and evaluation in basic intervention design

Intervention planning frameworks are used to map out links between action and results. Much has been written about different kinds of project and programme frameworks (e.g. logical framework and results frameworks) as key tools for planning and establishing monitoring and evaluation indicators. These frameworks do not need to be treated in detail here (see, for example, IFAD, 2002). What needs to be noted is that logical frameworks are very intervention-specific and cannot be prescriptive. There is no single model for a logical framework that can work for the very large range of climate-smart agriculture interventions, many of which will be nested within broader programmes and projects (see FAO, 2012a for additional guidance). 

A different organizing framework is the Driving forces - Pressure - State - Impact - Response (DPSIR) framework, which has been extensively applied in the context of environmental management. This framework follows a causal chain from the causes of an environmental issue (the driving forces), to its effect (impact) and required responses. Monitoring indicators are then identified in relation to the different elements of the causal chain. For an example, consult the FAO Land Degradation Assessment (LADA). The Theory of Change (TOC) approach similarly helps to lay out the broader cause and effect links in projects and programmes with several components. 

Project and programme frameworks are useful in delineating the expected outputs and outcomes that result from stakeholder participation. Indicators are more easily developed and organized around such a framework. The CCAFS Climate-smart Agriculture Programing and Indicator Tool (Box. C9.1) provides a guidance tool to help planners identify interventions and related indicators that can maximize the potential benefits related to the different climate-smart agriculture objectives (i.e. productivity and income, climate change adaptation and mitigation). 

The importance of tracking intervention processes 

It is important to distinguish between objectives that are oriented towards direct impacts and results ‘on the ground’ and those that are oriented toward processes. Even though an understanding of underlying processes is critical for climate-smart agriculture interventions, these processes are often neglected, as they are less easy to measure. Implementing climate-smart agriculture cannot be done in a strictly linear way from interventions to results. With rapid changes in the environment and the need to continuously address capacities for adaptation at both the institutional and household level, it is crucial to measure changes in processes and participation (Villanueva, 2010). For example, understanding why behavioural changes are taking place or not (Villanueva, 2010) is a process that is worthy of monitoring and evaluation. In this area, it is possible to draw on work from the broader agricultural development field (FAO, 2012b) and other disciplines.

Outcome mapping, developed by Canada’s International Development Research Center in a research context, has been adopted by a range of programmes for fostering institutional change. It is particularly helpful in delineating the expected outcomes among the different project partners and stakeholders. Outcome mapping is suited for monitoring institutional changes, capturing changes in capacity and the resulting delivery of services.

Ex-Ante economic and climate change analysis 

The basic design for large-scale programmes requires a sound financial and economic analysis to provide an economic justification for proposed interventions and gain a better understanding of their long-term sustainability. A financial and economic analysis, which examines the returns to costs at the farm and project level, provides input for decision-making regarding ex-ante project investment. Financial and economic and analyses can be used to complement specific climate change analytical tools, such as the Ex-Ante Carbon-balance Tool (EX-ACT). For more information on the use of EX-ACT in financing the adoption of climate-smart agriculture, see in chapter C3.5.3). It assesses the feasibility of reaching objectives and indicator targets for a range of possible interventions and their required resources and costs. This analysis helps ensure expectations remain realistic. See Module C8 on climate impact assessments and appraisals of climate-smart agriculture options.

Box C9.1  CCAFS Climate-smart Agriculture Programing and Indicator Tool

The CCAFS Climate-smart Agriculture Programing and Indicator Tool has been designed to address both the need for good instruments for programming and better metrics for tracking outcomes and impact. It also allows multiple development agencies and agricultural focused programmes to share a common framework on how they are addressing climate-smart agriculture, and how they can make their future programing process more climate-smart. The Tool guides users through a thoughtful and transparent process to:

  • examine to what extent its current intervention addresses each of climate-smart agriculture objectives (i.e. increased productivity and income, improved adaptation and, if possible, mitigation), or to what extent the planned intervention is climate-smart;
  • compare the scope and climate-smart agriculture intentionality among different project designs; and
  • support the identification and selection of an appropriate set of indicators to measure and track climate-smart agriculture outcomes.

By going through this climate-smart agriculture programming process, donors and implementers can:

  • provide visibility to climate-smart agriculture impact areas not originally targeted or focused by the intervention;
  • strengthen the planning phase of interventions to ensure that all outcomes (beyond increased productivity and income) are properly included in the monitoring and evaluation design; and
  • increase awareness on how to make their future interventions planning process climate-smart.

Supported by a database of over 378 indicators with climate-smart agriculture-related indicators gathered from several international development agencies and institutions, including FAO, The United Kingdom's Department for International Development (DFID), GIZ, IFAD's Adaptation for Smallholder Agriculture Programme (ASAP), the World Bank, and United States Agency for International Development (USAID), the Tool aims to facilitate the delivery of not only productivity outcomes, but also track adaptation and mitigation impacts.

The Tool consists of three steps:

Step 1: Definition of scope and intentionality of desired outcomes: By responding to specific questions related to the three climate-smart agriculture objectives, a traffic light system allows to specify the degree of intentionality desired of interventions (red: not at all, amber: indirectly and green: directly). The main objective is to enable users to more systematically check for potential co-benefits and/or unintended outcomes in more than one objective, and thus properly identify appropriate indicators and metrics for its monitoring and evaluation. 

Step 2: Selection of intended scale of action (household or farm, subnational, national) and the types of indicators based on the current stage of the intervention.

Step 3a: A results summary, which leads to a proposed set of relevant indicators that can be used to inform the design and monitoring and evaluation plan of future interventions.

Step 3b: Visualization: Intervention's evaluation through climate-smart agriculture lens and degree of intentionality.

Setting baselines and baseline projections

Baselines provide important data on the starting conditions against which the impacts of policy and programme intervention can be compared. This also includes capturing qualitative and quantitative baselines for system-wide capacity enhancement for climate-smart agriculture (See module C1 on system-wide capacity development). If climate impact assessments and climate-smart agriculture options assessments are carried out for a given point in time or for the projected future conditions, they constitute a baseline or baseline projection that is relevant for the evaluation of impacts of a particular policy, project or programme. These assessments can also be used for monitoring outputs and progress towards outcomes and impacts along the way. Examples of variables used to measure baselines are shown in Figure C9.3.

Figure C9.3.  Baselines and baseline projections

Based on climate impact assessments, baseline projections can be developed for expected future climate variations, the associated variations in agricultural outputs and respective vulnerabilities. These are projections of the impacts of climate change and the state of agriculture, food security and vulnerability without the programme or project’s interventions. Some common variables are listed in Table C9.1. This ‘without intervention’ scenario helps to frame broader policies and programmes (see Figure C9.1). Baseline projections can to some extent provide ‘counterfactuals’ and be used to evaluate the impacts of climate-smart agriculture and related interventions at a longer time scale than typical development projects. However, in a situation of complexity that climate change creates, the baseline can keep shifting. Climate impact predictions are just that – predictions – and they may not come true. During the course of the project, there may also be unexpected climatic events or other factors that make the old baseline irrelevant.

Detailed assessments of climate-smart agriculture options contribute to the development of more specific baselines of the current status against which measurements can be made to determine if climate-smart agriculture practices improve local agriculture. Structuring an inclusive stakeholder assessment to maximize country-ownership and commitment for mutually accountable results is discussed in module C1 on system-wide capacity development. These are project and programme baselines and usually refer only to the status at the beginning of an intervention. Associated with these baselines are the related indicators and targets of the expected objectives and changes, which are used to frame the climate-smart agriculture project. The evaluation of the impacts of climate-smart agriculture interventions is made against these project and programme baselines at the end of a project cycle. Progress being made towards important intermediate outcomes and results is also often monitored during implementation.

However, as climate conditions evolve over the project and programme cycles, and new information about impacts of and vulnerability to climate change becomes available, baseline projections may need to be revised periodically. Adaptation processes need to be designed to respond to evolving climatic conditions. The carbon balance of an ecosystem is dynamic and may change over time in the absence of mitigation interventions. Project managers may need to adjust the climate-smart agriculture interventions according to revised baseline projections at the project mid-cycle and evaluate the project’s benefits against the new projections. The World Agroforestry Centre and CCAFS baselines data collection activities described in boxes 9.1 and 9.2 provide robust information for both policy and programme comparisons over various timespans. 

Over a short period, changes in baselines or baseline projections can be subtle, so they are not a great concern for shorter climate-smart agriculture projects (GIZ, 2011a). The use of ‘control’ groups when doing impact evaluations should be able to account for some of the variability in baselines, as well as changes in other factors, such as markets and the broader economy.

However, for longer-term projects and programmes (more than 5 years), monitoring and evaluation should take place against a ‘moving’ baseline or up-to-date baseline projections as well as against the typical project and programme baselines. An additional use of baseline projections is recommended for climate-smart agriculture practitioners to deal with the characteristics of longer-term climate change adaptation and mitigation actions.

Table C9.1.  Examples of variables used for measuring baseline projections and project and programme baselines

Examples of variables used in climate change baseline projections to frame the initial context and situation within a specific geographic region include:

External variables:

  • key climatic variables, such as temperature, rainfall, and its seasonality;
  • frequency and intensity of extreme weather events;
  • climate-risk prone areas;
  • water availability;
  • the number of people affected by floods or prone to flood risks;
  • agricultural productivity in terms of crop yield without any adaptation measures; and
  • greenhouse gas emissions without any mitigation measures.

Examples of variables and indicators for setting project and programme baseline at the beginning of intervention to compare with end results include:

  • irrigation, water availability and withdrawal;
  • size of farm and land-use areas by crop (both cash crops and crops grown for household consumption) and management practices;
  • natural resources (e.g. watershed conditions, forest cover and fish stocks);
  • livestock numbers and management practices;
  • domestic market prices and their volatility;
  • population groups and their location categorized by poverty, food security, vulnerability and other key socio-economic factors, such as caste, class or age, disaggregated by sex; and
  • percentage of the population with access to and control over key resources for adaptation (e.g. climate-smart agriculture technologies, crop insurance, early warning information, seasonal climate forecasts) disaggregated by sex and other key socio-economic factors.

Developing indicators

Indicators are identified in the design stage, particularly when preparing a results-based framework. They are then further refined when there is greater clarity on the specific interventions and their expected scope of action within a programme or a project. Specific indicators for system-wide capacity development are discussed in module C1.

Table C9.2.  Examples of indicators of common outputs, outcomes and impacts in monitoring and evaluation for climate-smart agriculture programmes and projects  

The refinement of these indicators ensures that they are measurable and will be context specific. Where possible, disaggregating data (e.g. by gender and other key target groups) is extremely valuable.

Poverty and household impacts (where possible this data should be disaggregated by gender or by male- and female-headed households):

  • percentage of population that is food insecure;
  • percentage of population below the poverty line;
  • household income, income variability and diversification;
  • Gini coefficient;
  • marketing and commercialization chains that are adapted to changing conditions;
  • proportion of food and income that comes from climate-sensitive sources;
  • amount of time spent collecting firewood; and
  • amount of time spent collecting water.

Outcomes in terms of climate-smart agriculture-related changes in productivity:

  • agricultural productivity (e.g. tonnage of crop produced per hectare);
  • changes in land use in area;
  • reduced greenhouse gas emissions;
  • changes in productive resilience to climate variability;
  • multiple productive benefits across a range of production systems, resulting from synergies and links in system;
  • changes in biophysical characteristics (e.g. content of soil organic matter); and
  • diversification from climate-sensitive livelihood sources.

Outcomes in terms of adoption of climate-smart agriculture systems:

  • number of irrigation systems that raised drought prevention standards and area of farmland area covered by these systems;
  • number of soil and water conservation works;
  • area of farmland that adopted climate-smart agriculture technologies (e.g. reduced tillage, permanent crop cover, agroforestry);
  • forest area in which climate-smart technologies are adopted;
  • number of fisherfolk who adopted climate-smart fishery technologies, disaggregated by sex; and
  • increased access of women to land and/or productive resources.

Outputs and outcomes indicators related to capacity-development and service-related interventions:

  • number of people who benefited from capacity development, disaggregated by sex (output)
  • women beneficiaries constitute half of participants in capacity-development activities (output);
  • number of officials trained on the inclusion of gender issues in climate-smart agriculture (output);
  • number of male- and female-headed households that have gained direct household benefits from more climate-resilient agriculture infrastructure (outcome);
  • changes in farm-gate and market price (outcome); and
  • proportion of officials applying gained knowledge on gender issues in climate-smart agriculture (outcome). 

Institutional capacity development outputs and outcomes: 

  • strategies, policies and regulations formulated for climate-smart agriculture (output);
  • inclusion of climate change in agricultural policy frameworks (outcome);
  • actions identified and planned by local authorities to address significant vulnerabilities and opportunities not yet present in existing strategies and actions (output);
  • public commitments made to identify and manage climate-related risk (output);
  • proportion of budget allocated to support climate-smart agriculture (outcome);
  • proportion of budget allocated to agricultural research and development (outcome);
  • evidence of climate change mainstreaming in national and local agricultural development plans (outcome); and
  • increase in number of women participating in local, national and regional dialogues on climate-smart agriculture (output or outcome).

To measure project progress and achievements, it is necessary to identify suitable indicators and clarify related baselines, targets and means of verification for each of the results at different levels. This forms the core part of the project’s monitoring and evaluation framework. Indicators are extensively treated in monitoring and evaluation guides. Some key aspects of indicators in relation to climate change (see for example Brooks et al., 2011 and 2013) include characteristics, typology and range. 

Outcome indicators (or intermediary outcome indicators, depending on the terminology adopted by the donor) are mainly process indicators. In most climate change interventions, there is often a need to develop and establish outcome indicators to track, among other things:

  • system-wide capacity development across the enabling policy environment (e.g. aligned agricultural and environmental policies, demonstrated country-commitment through budget allocation), organizations and institutions (e.g. enhanced institutional coordination, clearly defined mandates, enhanced network and multistakeholder collaboration) and individual capacity of men and women (e.g. changes in attitude and behaviour to apply newly acquired knowledge);
  • infrastructure improvement with attention given to who has access to the improved infrastructure; and
  • technology dissemination, including technologies for climate change adaptation and mitigation in each of the agricultural sectors, and the uptake of this technology by men and women, as well as agricultural innovation systems and the uptake of these innovations.

Characteristics of indicators 

Indicators should, wherever possible, be Simple, Measurable, Attributable, Reliable and Time bound (SMART). The expanded set of SMART criteria presented below provides a useful guide for identifying appropriate indicators (modified from the Canadian International Development Agency (CIDA), cited in GIZ, 2011a):

  1. Validity: Does the indicator measure a change in climate risk or vulnerability?
  2. Precise and specific meaning: Do stakeholders agree on exactly what the indicator measures in this context?
  3. Practical, affordable, and simple: Are climate- and adaptation-relevant data actually available at reasonable cost and effort? Will it be realistic to collect and analyse information?
  4. Reliability: Can the indicator be consistently measured against the adaptation baseline over the short, medium and long term? With regard to mitigation, are the indicators robust enough for formal auditing under measurement, reporting and verification requirements?
  5. Sensitivity: When the respective climatic effects or adaptive behaviours change, is the indicator susceptible to those changes?
  6. Clear direction: Is it certain that an increase in value is good or bad and for which particular aspect of adaptation? Is it ultimately attributable to intervention?
  7. Utility: Will the information collected be useful and relevant for adaptive management, results accountability, and learning? Does it measure achievable results?
  8. Owned: Do stakeholders agree that this indicator makes sense for testing the adaptation hypothesis?

Typology of indicators

Using a simplified typology, indicators can be classified into four types. Each type of indicator is important for measuring outputs, outcomes and impacts in relation to climate change interventions. The four types of indicators are:

  • quantitative (e.g. tonnes per hectare of incremental crop production, number of days a year a household has adequate meals, or number of men and women with increased income);
  • qualitative (e.g. beneficiary perception of satisfactory service delivery by intervention agency);
  • proxy indicators that give an approximation of a desired measure in situations where a direct indicator is difficult to assess; and
  • indices, which are composed from other indicators to provide a more simplified aggregate measure of change.

Along with outcome-based indicators, it is also important to highlight process-based indicators. This is particularly relevant when capturing system-wide capacity development results (see module C1). Both types of indicators are important and have their own particular advantages and disadvantages. The outcomes and impacts will be heavily influenced by the capacities and practices of the institutions involved, the policies established, the level of intersectoral coordination, the resources available and other factors. Identifying these influences would be essential for any evaluation and for taking corrective action to modify the intervention and take any additional necessary steps. Villanueva (2010) propose the Adaptive, Dynamic, Active, Participatory and Thorough (ADAPT) framework with indicators that are more process-oriented. It is worth emphasizing the need to look at what has been put in place to strengthen adaptation and adaptive capacity outcomes. These outcomes are important even if events to test the adaptation may not take place during the intervention period (e.g. infrequent but stronger extreme events, such as large floods and hurricanes).

However, care must be taken when selecting pre-developed indicators for an intervention in a particular context. Many indicators designed by development organizations, while potentially useful at a local level, are sometimes more suited or intended for aggregating results across different projects, or presenting data in a simpler format for accountability to an organization’s senior management or for communications purposes to donors and other audiences. For the project team and beneficiaries to take ownership of monitoring and evaluation and its effectiveness, the indicators need to be meaningful to them, and customized to the local context, the particular challenges the project seeks to address, and the specific benefits it aims to bring. 

Range of indicators 

The objectives of climate-smart agriculture are to increase productivity and support the achievement of national food security and development goals, improve the ability of communities to adapt to the impacts of climate change, and where possible, to reduce or remove greenhouse gas emissions. Indicators for monitoring and evaluating projects’ impact should try to reflect these objectives. Some examples include:

  • agricultural productivity in the project area over a multiyear period (see GDPRD et al., 2008);
  • monitoring changes in land use on a wider scale, which can draw on literature, such as sustainable land management impact monitoring (Herweg et al., 2012);
  • resilience to flood and drought disasters over a multiyear period;
  • the total amount of annual greenhouse gas emissions reduced from the project areas over a multiyear period (see module A2 on climate-smart agriculture and mitigation);
  • prevalence of food security by household or by men and women in the project area over a multiyear period; and
  • the participation by key stakeholders, both men and women, in agricultural decision-making.

The guidance document Annex 6 in FAO 2012b provides a more detailed description and examples.

Indices capturing multiple outcomes of climate-smart agriculture 

The above range of indicators reflects the multiple objectives of climate-smart agriculture. See also Table C9.2  for examples of indicators of common outputs, outcomes and impacts in monitoring and evaluation for climate-smart agriculture programmes and projects. The World Bank, after a wide consultation process, has developed a set of indices for a number of indicators covering the areas of policy, technology, and results that are relevant to the project development and monitoring and evaluation cycle for climate-smart agriculture interventions. Box C9.4 summarizes these indices and examines how they have been tested and applied. Indices can provide an overview of the complex interventions that a project has achieved.  They can be used for making rough comparisons, provide accountability and enhance communications between other projects and programmes. However, at operational level, they may not be able to serve as a substitute for more specific measures and indicators that are required for project team learning and adaptive management.

Measurement of cross cutting changes

Although not an easy task, it is possible to measure variables that are financial in nature, such as income and assets. It is also possible to measure the benefits of climate change mitigation, for example, greenhouse gas emission reductions and increased soil carbon sequestration that can be translated into carbon dioxide equivalents. However, it is more complex to further translate reduced carbon dioxide emissions into economic values. This can be done using EX-ACT, which is based on a set of simple assumptions for estimating carbon dioxide reductions, in combination with various measures for accessing carbon funds (see module C4 on financial instruments). Measuring the outcomes for climate change adaptation is more difficult. There is no firm consensus on a set of measurable indicators at the outcome level. Inevitably, benefits can only be measured by more than one variable, which creates a situation where there is a risk of double accounting. In addition, many adaptation benefits, which are not traded as goods and services on markets, can only be valued using techniques from environmental economics.

Of particular importance in the context of climate change is the measurement of changes in vulnerability and resilience. In this regard, there are a number of indicators and indices that have been developed by FAO, the World Food Programme (WFP), non-governmental organizations and others (FAO, 2011; Frankenberger et al., 2012). Also, a considerable body of work exists on emergencies and disaster risk reduction and disaster risk mitigation. Twigg (2009) identified characteristics of disaster-resilient communities using indicators organized around components of resilience. These indicators are very specific to a particular group and area. In measuring outcomes, they may include specific household or community capacities to manage key natural resources and for measuring impacts, as well as key food supplies or household assets. For example, a household with savings and assets may be able to access funds in an emergency. FAO has assisted countries and partners by applying the Resilience Index Measurement and Analysis (RIMA) tool to provide a common set of metrics for comparisons over time and different locations. 

An important element will always be who decides on what interventions to implement and who benefits from them. For example, household decision-making about agricultural practices often has a strong gender dimension, with men and women taking responsibility for different spheres of influence. Sometimes, decisions made by one group affect another group that has had no say in the matter (e.g. men may choose a crop or practice that earns more income for the household, but increases the amount of time women spend weeding or watering). In other cases, men and women take joint decisions, particularly in times of crisis. The collection of data disaggregated by sex and beneficiary groups is crucial for measuring these changes. The gender dimensions of climate-smart agriculture are addressed in module C6. Tools for data collection to assist in integration gender in climate-smart agriculture are covered in greater detail in FAO 2016. Approaches to formulating gender-sensitive indicators is given in Box C9.3. 

Box C9.3  A gender-responsive approach to climate-smart agriculture evidence and guidance for practitioners 

Indicators focusing on gender issue within a project include: 

  • number of gender-responsive technologies developed by research activities that are applicable under a climate-smart agriculture approach. 

Gender-responsive technologies are defined as: 

  • technologies based on needs and interest of both female and male farmers; 
  • technologies that reduce time and labour for women farmers, and 
  • technologies that are accessible and affordable to both men and women. 

Indicators of project outcomes designed to capture information on men and women to analyse the gender-related impacts include: 

  • number of farmers who have access to and use weather and climate information services and price information on a regular basis (disaggregated by sex). 
  • percentage change in crop yield per hectare and year as result of the climate-smart agriculture intervention (disaggregated by male or female-headed households and household members). 
  • number of farmers participating in functional associations as a result of the project (disaggregated by sex and by type of association, for example, market cooperatives, producer associations). 
  • farmers who consider themselves better-off (e.g. in terms of livelihood, income, nutrition, well-being, social status or empowerment) now than before the climate-smart agriculture intervention (disaggregated by sex). The measurement of this indicator would require direct feedback from farmers through a survey.

Source: FAO, CGIAR, CCAFS, 2016.

Box C9.4  World Bank Climate-Smart Agriculture Indices 

The climate-smart agriculture indices are based on a range of climate-smart agriculture indicators in the areas of policy, technology and results. The development of the climate-smart agriculture indicators was informed by an encompassing climate-smart agriculture impact pathway that traces how project outputs can result in behavioural change (project outcomes). The climate-smart agriculture indicators aim to capture direct project outputs and behavioural changes from a range of stakeholders, such as producers, policy makers, and civil society. Behaviour change is seen as a determining factor because only when a key group of stakeholders has changed their behaviour can the impacts achieved through a climate-smart agriculture intervention be sustained. The participatory and rigorous selection allowed for the development of a comprehensive set of indicators that can potentially provide the empirical basis for identifying viable climate-smart options, select contextually relevant technologies and practices, monitor results, and assess policies and the necessary enabling activities for climate-smart agriculture.

There are three climate-smart agriculture indices: the climate-smart agriculture Policy Index (climate-smart agriculture -Pol Index), the climate-smart agriculture Technology Index (climate-smart agriculture -Tech Index), and the climate-smart agriculture Results Index (climate-smart agriculture -Res Index). The climate-smart agriculture -Pol Index is established on the national level and measures a country’s institutional readiness to support climate-smart agriculture interventions. The climate-smart agriculture -Tech and climate-smart agriculture -Res Indices are applied on the project level. The climate-smart agriculture -Tech Index serves as an ex ante measure of the ability of climate-smart agriculture interventions to reach the climate-smart agriculture's three objectives. The climate-smart agriculture -Res Index can be applied to measure a project’s success to reach of climate-smart agriculture's objectives.

The climate-smart agriculture Policy Index comprises three themes, 14 indicators, and 31 subindicators. The first theme, 'readiness mechanisms', refers to the capacity of countries to plan and deliver adaptation, mitigation, economic readiness, governance readiness, and social readiness programmes in ways that are catalytic and fully integrated with national agricultural development priorities. The second theme, 'services and infrastructure', reflects the ability to leverage agricultural investments through the provision of services and an enabling environment. This includes extension services, research and development, roads, social safety nets, greenhouse gas inventories, risk management systems, and adaptive capacities. The third theme, 'coordination mechanisms', assesses collaboration for disaster risk management and the coordination among sectors involved in climate-smart agriculture. The climate-smart agriculture policy indicators enable policy makers and other users to gauge how a country’s enabling environment for climate-smart agriculture is changing over time. They are also useful in identifying gaps in the implementation of climate-smart agriculture activities and in developing benchmarks for reform.

The climate-smart agriculture Technology and Practices Index comprises 27 indicators clustered into three main themes: productivity, resilience and mitigation. Ex ante application of the index reveals how project interventions can lead to productivity gains and environmental benefits. It is particularly useful in identifying the most appropriate technologies for a climate-smart agriculture project during its planning and design stages.

The climate-smart agriculture Results Index comprises 22 indicators, clustered in three categories and eight topics, intended to help project leaders measure an agricultural project’s performance toward achieving the objectives of climate-smart agriculture. The three categories have been identified according to whether the indicators measure direct output of a climate-smart agriculture project intervention, the climate-smart agriculture enabling environment, or the medium- to long-term outcomes of a climate-smart agriculture intervention. The eight topics include beneficiaries, land use and land cover, livestock, the enabling environment, natural resources, emissions, yields, and benefits and welfare. In addition, the indicators are assigned to meet climate-smart agriculture's three objectives: productivity, resilience and mitigation. The climate-smart agriculture -Res Index can be applied to measure the project’s performance after project completion and during project implementation. It also gives project teams the flexibility to customize the index and adjust it to their specific context.

These indices constitute a solid framework for presenting project achievements in a general form or at a high level, or for aggregating results from a whole portfolio of interventions and showing the patterns of results across them. They represent a well-structured set of higher-level indicators that can be used for accountability and communications purposes. Under this framework, more context-specific indicators for individual projects, which are needed for project team learning and adaptive management, can be developed. 

Source: World Bank 2016.

C9 - 2.3 Methods in monitoring and evaluation

This section provides an overview of the range of monitoring and evaluation methods applied to climate-smart agriculture programmes and projects, which are generally not different from the standard range of methods and tools. For more information in this area, consult the FAO Investment Learning Platform on Monitoring and Evaluation. These methods refer to the implementation phase of the monitoring and evaluation process, but also need to be considered during the preparation phase.

Methods for measuring results of outcomes and impacts (mainly for evaluation) and progress against expected targets (mainly for monitoring) are substantial topics that are best examined in relation to literature on specific indicators and the broader literature on monitoring and evaluation. Many indicators related to implementing climate-smart agriculture deal with well-established measures of change (e.g. technology adoption, land-use change, household livelihoods and institutional change) that have been dealt with extensively in development literature and through a range of methods and good practices. 

Methods for programme and project preparatory assessment for Monitoring and Evaluation 

Technical aspects of baselines are addressed in module C8 on climate impact assessments and appraisals of climate-smart agriculture options and module C1 on system-wide capacity development. Here it is important to note that methods and process for baseline assessments ideally need to consider the type of information that will be required for doing robust end-of-project evaluations. 

Project and programme baseline assessments are done through surveys in intervention and control areas. The surveys, which measure food security, livelihoods and agriculture systems, incomes, basic household assets and services, and environmental parameters, have to be designed with appropriate sampling to ensure that useful end of project analysis can be done.

Participatory poverty and vulnerability assessments help determine who are the most vulnerable groups in the community as defined by community members’ own criteria. This serves to identify key intervention target groups, and assesses how these groups have may have changed as a result of climate-smart agriculture interventions.

Participatory stakeholder, capacity, institutional and legal assessments consider the starting situation and needs for changes in capacities, human resources, organizational systems, institutional coordination, and laws and policies (See module C1 on system-wide capacity development). 

Methods for project monitoring 

Monitoring needs to have in-built and integrated systems for tracking financial transactions, expected outputs, activity targets and achievements. Monitoring must also incorporate feedback and learning into programme and project management. To track and manage data, many of these monitoring systems make use of computerized management information systems. A critical element of successful monitoring and evaluation is the internalization of its importance in planning and decision-making by management and other stakeholders. Monitoring and evaluation tasks are too often merely seen as reporting for governments or donors. Basic methods and tools for monitoring are:

  • Regular project monitoring involves the gathering of activity and output progress data, financial management information, and signalling emerging issues or good practices.
  • Management information systems are web-based support systems that are increasingly being managed through remote devices, linked to financial management (FMS) and geographic information (GIS) systems. FMS and GIS take on a greater importance for climate-smart agriculture as they may be linked to accounting for payments for environmental services, greenhouse gas accounting and accurate measurements of spatial changes in natural resources.
  • Agriculture and natural resource management monitoring is carried out at frequencies and scales significant enough to provide meaningful information. The measurements can be done by a range of methods from structured crop to participatory transect walks.
  • Process monitoring is often done to support regular monitoring, assess project process and institutional changes and relationships, and rapidly identify management responses. If sensitively and rigorously done, often by an external third party with independent insights, it can provide a powerful tool for management to quickly learn about capacity issues, the changes that may be required, and whether participation mechanisms are working, and respond appropriately. This is particularly relevant for climate-smart agriculture interventions, which follow new development approaches, need to be flexible, and are looking for ways to scale up their activities.
  • Participatory monitoring and evaluation methods include a wide range of methods for enhancing the engagement of the communities involved in programmes and projects, often as beneficiaries and partners, in gathering information, and for increasing their ownership and adoption of the project goals.

Methods for evaluation

Evaluation, at key milestones in the programme, but especially at its conclusion or after its completion, involves gathering specific data relating to project indicators, and focuses on outcomes and impacts. It is important to revisit these indicators using the same methods for collecting data and information that were employed in the baselines assessments. However, evaluation is not only about the impact evaluation surveys. It also assesses the ongoing relevance of the design, objectives and implementation mechanisms; reviews the efficiency, performance of different actors; and highlights lessons learned. Much of this evaluation work consists of reviewing existing reports and data, interviews, small studies and consultations. 

Impact evaluation methodology

Impact evaluation assesses the impact of an intervention using counterfactual analysis. The estimated impact of the intervention is calculated as the difference in mean outcomes between a ‘treatment group’ (i.e. those receiving the intervention) and a ‘control group’ (i.e. those who don’t). This is done through randomization (experimental design), pipeline and matching. To date, impact evaluation has not been used extensively in the climate change context, and it faces some challenges in this area, due to the scales involved, the externalities and process orientation. Nevertheless, impact evaluation is being increasingly advocated to understand the attribution of impacts. It has been used in mitigation programmes, and is often applied in agricultural and rural development projects, and natural resource management projects (Prowse and Snilstveit, 2009). Given the complexities involved when working under changing climatic conditions and the need to constantly adapt the project and practices to deliver benefits to smallholder agricultural producers, finding a true counterfactual becomes nearly impossible and may not add much real value.

As mentioned earlier, a particular methodological issue regarding impact evaluation is the importance of attribution. The challenge of attribution arises when attempting to ascribe observed changes and results specifically to a project, in cases where these changes could also be due to other external factors and interventions. This is a major concern for climate change programmes since they are potentially affected by long-term and large-scale climate and economic processes. In the context of projects, the issue is addressed through the design of rigorous project baselines and impact evaluation surveys, which take into account external effects. These activities include ‘control’ areas and households in the survey samples, against which changes in land use and the livelihoods of the project's beneficiaries’ can be compared. As mentioned above, impact evaluations can be challenging for climate change interventions. There are also other factors to consider, which add to the complexity. For example a given project site is situated within a larger landscape, and this landscape, which affects ecosystem dynamics in the project site, may not be taken into consideration in the impact study. Also, with large-scale projects involving several thousand households it is difficult to find similar households that have not been affected by the project given local political, market or social dynamics. Impact evaluations are also more costly. However, opportunities may be found for impact evaluations, and these opportunities should be taken advantage of. 

Contribution analysis, which is the predominant approach of international development evaluations, can be valuable (Mayne, 2008). Under this approach, the causality behind an outcome from an internal or external factor, or earlier outcome, is inferred using evidence. Rather than saying “A is attributable to B” (based on experimental methods), one would say, using contribution analysis, that “it can be plausibly concluded, with some level of confidence, that B contributed to A.” However, this approach depends upon an articulated and fixed theory of change, and climate change may require constant reformulations of the theory of change. Nevertheless, making inferences based on evidence on an ongoing basis and in real time might help ensure that the project is meeting climate-smart agriculture goals.

Developmental and utilization-oriented evaluation

The standard approach for evaluating projects and programmes, involves conducting a formative, or mid-term, evaluation to assess whether the intervention model is working and gauge the results achieved to date. A summative evaluation is carried out at the end of the project to determine whether the objectives were achieved and to draw lessons. These evaluations have proven to be useful. The OECD/DAC evaluative criteria, applied in or adapted for many international development evaluations, have succeeded in drawing out key findings of projects and programmes. However, these interventions, as complex as some may be, have been regarded as linear, straightforward and fixed in nature. Developmental evaluation, on the other hand, involves using evaluative thinking and evidence on an ongoing basis during the intervention to assess the results of innovations and learn by doing, and guide decision-making by stakeholders in an environment of complexity. The 'model' never stops developing. This makes monitoring and evaluation more useful, since it not only gauges project effectiveness, but also identifies in real time what is and is not working, or changing, for communities, and what adaptations are needed. 

Developmental evaluation also constitutes a form of utilization-oriented evaluation. A standard evaluation’s consultation with stakeholders at the design stage and its lessons for them at the end can be a useful exercise. However, a utilization-oriented evaluation that goes well beyond accountability, “requires moving from the general and abstract, that is, possible audiences and potential uses, to the real and specific: actual primary intended users and their explicit commitments to concrete, specific uses” (Patton, 2011). Financial and economic analyses, which use mainly agricultural, environmental and socio-economic data, as well as detailed market, labour and trade information, consider the economic and financial returns at the household, farm and system levels. This is most often used at the design stage for feasibility assessment, and for an evaluation of the final economic returns at end of the project. However, financial and economic analyses can be used also for ongoing assessment of the viability of economic activities related to climate-smart agriculture. 

Monitoring and evaluation system-wide capacity development for climate-smart agriculture

Module C.1 on system-wide capacity development provides further insights into the importance of tracking system-wide capacities for individuals, organizations and institutions and throughout the enabling policy environment to facilitate the transition towards climate-smart agriculture. It also describes the key mechanisms and tools for enhancing monitoring capacities to foster country-ownership and commitment. It is important to complement the socio-economic and institutional monitoring and evaluation elements described above, with activities that collect information and measurements on the programme's expected aims and results with regard to applied knowledge, effective organizations, institutions, networks and an enabling policy environment that is committed to promoting the uptake of climate-smart agriculture practices. Module C.1 outlines the process to establish a qualitative and quantitative baseline for a proposed capacity assessment for climate-smart agriculture. This process incorporates FAO normative work that has been published in the Organization's Capacity Development Learning Modules, particularly module C1 on Capacity Development Programming: Processes and Tools, which provides a guide to assess existing capacities, set priorities and objectives for interventions, track capacity development results and ensure that these are sustained and scaled up by national and/or regional institutions. Done jointly with stakeholders, this process maximizes country-ownership and commitment. The methods and tools cover:

  • joint analysis of the context and assessing capacity needs for individuals, organizations and institutions and throughout the enabling policy environment using problem trees, stakeholder mapping and analysis, and review of drivers of change;
  • measuring capacity development ('what' and 'how') by providing guidance on formulating capacity-development outputs, outcomes and indicators, outcome mapping , developing monitoring and evaluations plans for capacity change, and knowledge, attitudes and practices (KAP) surveys; and
  • building sustainability using checklists for sustainability, concerted capacity-building components and exit strategies within projects and programmes.

Above all, it is important to involve the stakeholders throughout the process to foster learning and maximize country-ownership and commitment to achieve the identified results. Particular attention needs to be given to the 'jump' from output to outcomes to ensure that skills and knowledge are being applied, organizations are performing better, policies are being implemented. 

Participatory monitoring and evaluation and stakeholder engagement

Efforts to address the impacts of climate change unite various sectors and stakeholders. Climate-smart agriculture requires reinforcing and extending the dialogue and engagement of key players. In the context of climate change, a typical agricultural development project can be relevant for a wide range of issues, including public health and migration. Climate-smart agriculture will not be successful unless all stakeholders are engaged at all levels and across all relevant sectors (UNFCCC, 2010; Hedger et al., 2008; Villanueva, 2010; GIZ, 2011a). The stakeholders to be involved vary from local communities to international organizations (Hedger et al., 2008). Ideally a landscape or territorial approach,  which encompass key ecosystems and communities engaged and affected by planned interventions, should be followed. In general, stakeholders must be representative of the target population. Men and women from different socio-economic groups should be involved, and different actors along the value chain should be consulted. Integrated frameworks for climate-smart agriculture across all levels will allow for clear and effective feedback mechanisms.

Especially with regards to mitigation, national-level monitoring and evaluation may need to be linked to UNFCCC requirements for indicators, such as greenhouse gas inventories and measuring, reporting and verification requirements for reduced emissions from deforestation and forest degradation (REDD+), At the project level, monitoring and evaluation should ideally be based on a national annual planning, monitoring, evaluation and budgeting system that provides orientation and harmonizes different projects under national programmes and policies for poverty alleviation, natural resources management and agricultural development.

Participatory, gender-responsive approaches in assessment, monitoring and evaluation, which recognize that local men and women are best suited to understand the local agricultural conditions, should be promoted. In these approaches, men and women contribute to assessing the impacts of climate change on their livelihoods and food security, and identify and measure their own indicators of change for monitoring and evaluation. This creates opportunities to develop a learning partnership that involves all the implementing partners and the participating communities. Participation in the assessment by the communities that are affected is critical, not only for gathering as much information as possible on the local situation, but for building ownership and commitment to the process by the community and increasing the likelihood of successful implementation of climate-smart agriculture practices.

Participatory planning, monitoring and evaluation have been recognized as important in the broader development context. They are used widely in rural community development interventions, and are particularly helpful for understanding community perspectives on the opportunities and constraints associated with climate-smart agriculture; dealing with fine-scaled variability in changes in climate and other factors; and empowering local communities to engage in community-based natural resources management (see Guijt, 1999 for methodologies, and FAO, 2008 for an example of adaptation to drought through community groundwater monitoring). The ADAPT process proposed by Villanueva (2010) also assists in organizational learning, monitor perceptions and promote organizational reflection and change. 

Community participation becomes essential for adaptive management and developmental evaluation under complexity. It contributes to building the capacities of beneficiaries to gather data and engage in evaluative thinking. Participation and ongoing feedback from communities is the sole means by which the intervention, and government institutions involved in programme implementation and research in the longer run, can receive information from the field on how climate-smart agriculture practices are performing in terms of their three main objectives and what the trade-offs appear to be. In developing countries, establishing or strengthening farmers’ organizations has been an important activity for a number of reasons. Under changing climatic conditions, and from a broader systems perspective, farmers' organizations will need to assume the additional role of drawing conclusions on the practices they have applied and communicating their findings not only to project managers to help assess the project's results and identify needs for adaptation, but to research institutions to develop the knowledge base in the emerging climate change field. Similarly, agricultural extension services, in addition to promoting the dissemination and exchange of technology to support adaptation to climate change (Simpson and Burpee 2014), can also assist in gathering the data required for monitoring and evaluation, and transferring this information to research institutions. This will require greater coordination between agricultural extension services and farmers’ organizations, as well as between the different sectoral line departments working at the landscape level. In most countries, fostering this type of coordination has been problematic. 

Supporting good practices in monitoring and evaluation

Experience has shown that the above activities (e.g. surveys, reporting, capacity development for monitoring and evaluation) may be implemented mechanically without being of great use for decision-making. Some useful practices in relation to strengthening monitoring and evaluation are listed below.

  1. Triangulation and mixed methods are necessary. No one tool will provide all the needed information, and the complementary use of tools, quantitative and qualitative methods will be important. There can also be difficulties in applying a strictly experimental design due to resource constraints and ethical concerns.
  2. Obtaining management and stakeholder ownership and engagement in the monitoring and evaluation process is important. Often monitoring and evaluation is seen as a stand-alone reporting task. There is a need to internalize monitoring and evaluation into project management and staff responsibilities.
  3. There is also a need to enhance capacity across the board on planning and monitoring and evaluation, and create a learning culture. Climate-smart agriculture projects are often experiments, and need to maximize opportunities for learning lessons that can be used to potentially scale up activities.
  4. Networks and platforms for sharing experiences, lessons learned and practical knowledge between projects and programmes have been for improving monitoring and evaluation. Some examples of networks on monitoring and evaluation are the International Initiative for Impact Evaluation (3ie) and Monitoring and Evaluation News. Similar networks and platforms have considerable potential for monitoring and evaluation in climate-smart agriculture.
  5. Participatory monitoring with stakeholders to foster ownership, commitment and mutual accountability for results is crucial.

Examples of tools for supporting monitoring and evaluation in climate-smart agriculture projects

Tools that are particularly important for monitoring and evaluation of climate-smart agriculture are those that address vulnerability, climate change mitigation, and the monitoring of resources. Some examples of practical tools have been used by FAO and other support agencies to assist countries in this area are listed below. 

FAO has developed and assisted countries and partners in applying the FAO Resilience Index Measurement and Analysis (RIMA). Measuring resilience is challenging, since it is multidimensional and cannot be observed or quantified directly. RIMA is a quantitative approach that allows for a rigorous analysis of how households cope with shocks and stressors. The results from resilience analyses using RIMA provide the necessary evidence to improve the design, delivery, monitoring and evaluation of the assistance offered to vulnerable populations, based on what they need most. RIMA methodologies estimate resilience through a set of pillars, which are then aggregated through latent variable models. Comparisons can be made between different types of households (e.g. male-headed households versus female-headed households or urban versus rural) in a given country or area. RIMA measurements help to answer questions such as: who is most in need? where should investment focus in terms of geographical location? which dimensions of resilience need to be supported? to what extent have interventions increased or decreased the target populations’ resilience?

EX-ACT provides ex-ante estimations of the impact of agriculture and forestry development projects on greenhouse gas emissions and carbon sequestration. EX-ACT is a land-based accounting system, measuring carbon stocks and stock changes per unit of land, expressed in tonnes of carbon dioxide equivalent per hectare and year. The ex-ante carbon balance appraisal guides the project design process and decision-making on funding aspects and complements the usual ex-ante economic analysis of investment projects. EX-ACT has the potential to support project designers in selecting project activities with higher benefits both in economic and climate change mitigation terms. EX-ACT outputs can also be used in financial and economic analysis. 

Technical expertise in forestry has been provided for direct support to individual developing countries in the design and implementation of National Programmes for the UN-REDD Programme and targeted support upon request on thematic areas, such as National Forest Monitoring Systems and measuring, reporting and verification (MRV) systems (See module B3 on climate-smart forestry). This technical expertise assists countries in developing their capacities to build robust national monitoring forest systems for REDD+ that also provide broader monitoring needs for sustainable development. It supports countries on how to gather information on safeguards, design safeguard information systems, and carry out governance assessments.  Countries are also given advice on land tenure regimes and legal preparedness and identification of best practices for sustainable forest management. Participatory approaches where forest resource users are directly involved in monitoring programmes can increase efficiency of data collection and buy-in to programmes (see boxes 9.7 and 9.8). A strong MRV system can be also combined with EX-ACT tools for example for national development banks to make assessments on the feasibility of investing in agriculture and forestry development, which both reduce emissions and increase resiliency (Bockel et al., 2016). 

Support for rapid remote sensing has been developed with Collect Earth, a FAO-Google collaboration for monitoring and evaluation that uses existing open access remote sensing data. It has been used for national forestry assessments. Collect Earth has also been applied to measuring programme and project baseline situations and evaluating impacts, for example in the ex-post assessment of land-use changes comparing project and non-project control areas.

Box C9.7  Community monitoring and national measuring, reporting and verification

The Forest Carbon Partnership Facility has looked at the advantages of community monitoring compared to expert monitoring for national REDD+ MRV. When communities are trained to use standard forest inventory protocols for carbon stocks using the Intergovernmental Panel on Climate Change (IPCC) recommended procedures, their monitoring can be reliable and more economical than expert inventories. Engaging communities in monitoring activities strengthens their rights and their stake in REDD+. The ownership of the data remains with communities, which increases their motivation for supporting REDD+. Data collected by communities can also be used in stock assessments in national forest inventories and support the information gathered at basic grid points.

Box C9.8   MRV in an agricultural carbon project in Kenya

Vi Agroforestry (Vi-skogen) is a Swedish development cooperation organization that works with farmers in the Lake Victoria Basin in Eastern Africa. The carbon project, which targets 60 000 smallholder farmers over 45 000 hectares in Western Kenya, plans to generate verified emission reductions through sustainable agricultural land management practices. MRV requires direct, activity-based measurements that include estimates of tree carbon, which is measured by the diameter of trees at breast height and their allometric growth functions, and soil carbon, which is modelled based on crop yields and land management practices data. Every year, all project participants collect core datasets. In addition, a sample of 200 farmers collect more intensive data. Global positioning system (GPS) units are used to measure farm plot size and location. However, most of the data are collected manually with pencil and paper. The project maintains two data management systems with datasets on livelihoods and carbon. The database automatically checks the quality of manually entered data and calculates mitigation impacts. The carbon accounting methodology has been approved by the Verified Carbon Standard and is in the public domain. The project is working to develop a cost-effective way of monitoring emission reductions that minimizes transaction costs and maximizes benefits to farmers.

Source: IFC, 2012; PwC, 2012