Project Implementation (Cont.)

12-Methodological Framework for Project Evaluation

International Fund for Agricultural Development (IFAD)

A framework for the systematic evaluation of development projects.

When can the Methodological Framework for Project Evaluation help?

Project impacts are not identified and cannot be attributed to project activities
Project evaluations do not assess the relevance of project objectives
There is inadequate data to support quantitative assessment of project efficiency
The contributions of different partners to development projects are not transparent
Project evaluations do not consolidate gender-related impacts
Project evaluations cannot be compared across similar or related projects
Lessons that could be used to improve future projects are not identified.

What can the Methodological Framework for Evaluation be used for?

Evaluating project performance and project impacts
Assessing project partners’ contributions to development projects
Providing a basis for accountability for project impacts
Identifying and documenting lessons to inform future, related projects.

Why use the Methodological Framework for Evaluation?

Take a systematic approach to evaluation using a clear set of evaluation criteria
Aggregate evaluation results across and within projects
Assess project impacts at project completion
Consolidate the evaluation of results, impacts, project performance and lessons
Focus on sustainability, innovation and the replication of project results and the impacts on gender equality and women’s empowerment.

Key concepts

Impact (2): For the purposes of the Methodological Framework for Project Evaluation (IFAD's purposes), impacts are the intended or unintended changes in the lives of the rural poor - as perceived at the time of the evaluation - to which the organization's interventions have contributed.

Impacts are determined at the end of project implementation and not some time later when secondary and indirect effects may be observed. They include what, in other evaluation frameworks, may be called results, outcomes and effects.

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Outline of the Methodological Framework for Project Evaluation

In the Methodological Framework for Project Evaluation, project achievements and impacts are assessed against key evaluation criteria. The criteria and notes on their use are outlined in Table 12.1 and more detail is provided in the following text boxes.

Table 12.1: Evaluation Criteria

1. PROJECT PERFORMANCE
1.1 Relevance of project objectives	The extent to which the project objectives (at the time of evaluation) are consistent with: the rural poor's perceptions of their needs and potential at the time of the evaluation; the economic, social and policy environment; the organization's mandate, strategic framework and policies; and the country's current poverty-reduction policies and strategies.
1.2 Effectiveness	The extent to which the project's major objectives (at the time of evaluation) were achieved, or are expected to be achieved, at project completion. Project effectiveness is measured with respect to the project designer's expectations of project's impacts (see Tables 12.1 and 12.2).
1.3 Efficiency	The extent to which the project achieved, or is expected to achieve benefits commensurate with inputs, based on economic and financial analysis or unit costs compared with alternative options and good practices to determine how economically resources have been converted into results (see Box 12.2).
2. IMPACT ON POVERTY (see Table 12.2 for key evaluation questions)
2.1 Impact on physical and financial assets
2.2 Impact on human assets
2.3 Impact on social capital and people's empowerment
2.4 Impact on food security
2.5 Impact on the environment and communal resource base
2.6 Impact on institutions, policies and the regulatory framework
2.7 Overarching factors: (i) sustainability - assess whether the net benefits of the project can be maintained in the long term (see Box 12.3)
(ii) Innovation and replicability/scaling up - assess the catalytic role of the organization or project in developing cost-effective ways to address problems/opportunities faced by the rural poor (see # 10 - Technology transfer for SARD)
(iii) Impact on gender equality and women's empowerment - assess the gender-related impacts in relation to each impact domain to consolidate gender considerations (see Box 12.4)
3. PERFORMANCE OF PARTNERS (see Box 12.5 for key evaluation questions)
3.1 Primary organization
3.2 Cooperating institutions
3.3 Government agencies
3.4 Non-governmental organizations and community-based organizations
3.5 Co-financiers

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Table 12.1: Project Effectiveness Matrix (completion of this matrix requires prior assessment of project impacts - see Table 12.2)

MAIN DOMAINS OF IMPACT	Key Questions for Impact Assessment(Changes to which the project has contributed)	Expectation of Impact(Project's Stated Objectives)				Effectiveness Rating(Achievement against stated Objectives - 4/3/2/1)
MAIN DOMAINS OF IMPACT		Reach who?	Change what?	Change how much?	Reach how many?	Reach who?	Change what?	Change how much?	Reach how many?
1. Physical and financial assets	1.1 Did farm households' physical assets change (land, water, livestock, trees, equipment etc)?
	1.2 Did other household assets change (houses, bicycles, radios etc)?
	1.3 Did infrastructure and people's access to markets change (transport, roads, storage, communication facilities etc)?
	1.4 Did households' financial assets change (savings etc)?
	1.5 Did rural people's access to financial services change (credit, saving, insurance etc)?
2. Human assets	2.1 Did people's access to potable water change?
	2.2 Did access to basic health and disease prevention services change?
	2.3 Did the incidence of HIV infection change?
	2.4 Did maternal mortality change?
	2.5 Did access to primary education change?
	2.6 Did primary school enrolments for girls change?
	2.7 Did women's and children's workloads change?
	2.8 Did adult literacy rate and/or access to information and knowledge change?
3. Social capital and people's empowerment	3.1 Did rural people's organizations and institutions change?
	3.2 Did social cohesion and local self-help capacity or rural communities change?
	3.3 Did gender equality and/or women's conditions change?
	3.4 Did rural peoples feel empowered vis-a-vis local and national authorities and development partners? Do they play more effective role in decision making?
	3.5 Did rural producers feel empowered vis-a-vis the marketplace? Are they in better control of input supply and marketing of their products?
4. Food security (production, income and consumption)	4.1 Did children's nutritional status change?
	4.2 Did household food security change?
	4.3 Did farming technology and practices change?
	4.4 Did the frequency of food shortages change?
	4.5 Did agricultural production change (area, yield, production mix etc)
5. Environmental and common resource base	5.1 Did status of the natural resource base change (land, water, forest, pasture, fish stocks etc)?
5. Environmental and common resource base	5.2 Did exposure to environmental risks change?
6. Institutions, policies and regulatory framework	6.1 Did rural financial institutions change?
	6.2 Did local public institutions and service provision change?
	6.3 Did national/sectoral policies affecting the rural poor change?
	6.4 Did the regulatory framework affecting the rural poor change?

Source: IFAD, 2003. A Methodological Framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome SARD Project Toolkit

Table 12.2: Impact Matrix

MAIN DOMAINS OF IMPACT	Key Questions for Impact Assessment (Changes to which the project has contributed)	Assessment of Change				Reach of Change
		Presence /direction of change (+)(0)(-)	What has changed? (Indicator)	Extent of change:		How many? (house-holds/ people)	Who? (poor/ poor-est/ better off)	Who? (M/F)	Project contribution 4/3/2/1	Dynamic processes 4/3/2/1**	Sustain-ability potential 4/3/2/1***
		Presence /direction of change (+)(0)(-)	What has changed? (Indicator)	How much?	Rating (4/3/2/1)*	How many? (house-holds/ people)	Who? (poor/ poor-est/ better off)	Who? (M/F)	Project contribution 4/3/2/1	Dynamic processes 4/3/2/1**	Sustain-ability potential 4/3/2/1***
1. Physical and financial assets	1.1 Did farm households' physical assets change?
	1.2 Did other household assets change?
	1.3 Did infrastructure and people's access to markets change?
	1.4 Did households' financial assets change?
	1.5 Did rural people's access to financial services change?
2. Human assets	2.1 Did people's access to potable water change?
	2.2 Did access to basic health/disease prevention services change?
	2.3 Did the incidence of HIV infection change?
	2.4 Did maternal mortality change?
	2.5 Did access to primary education change?
	2.6 Did primary school enrolments for girls change?
	2.7 Did women's and children's workloads change?
	2.8 Did adult literacy rate/access to information/knowledge change?
3. Social capital and people's empowerment	3.1 Did rural people's organizations and institutions change?
	3.2 Did social cohesion/local self-help capacity/communities change?
	3.3 Did gender equality and/or women's conditions change?
	3.4 Did rural peoples feel empowered vis-a-vis local and national authorities and development partners? Do they play more effective role in decision making?
	3.5 Did rural producers feel empowered vis-a-vis the marketplace? Are they in better control of input supply and marketing?
4. Food security (production, income and consumption)	4.1 Did children's nutritional status change?
	4.2 Did household food security change?
	4.3 Did farming technology and practices change?
	4.4 Did the frequency of food shortages change?
	4.5 Did agricultural production change (area, yield, production mix)?
5. Environment and common resource base	5.1 Did status of the natural resource base change?
5. Environment and common resource base	5.2 Did exposure to environmental risks change?
6. Institutions, policies and regulatory framework	6.1 Did rural financial institutions change?
	6.2 Did local public institutions and service provision change?
	6.3 Did national/sectoral policies affecting the rural poor change?
	6.4 Did the regulatory framework affecting the rural poor change?

* Rating: 4=high, 3=substantial, 2=modest, 1=negligible (based on rural poor’s perspectives in relation to the baseline situation)

** This refers to cases where even though impact achievement is modest or negligible, the project has set in motion dynamic processes that will eventually lead to substantial impact achievement. The identification of these processes is left to the evaluator’s judgment on a case-by-case basis.

*** Rating: 4= highly likely, 3=likely, 2=unlikely, 1=highly unlikely Source: IFAD, 2003. A Methodological Framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Box 12.2:		Qualitative assessment of efficiency
Qualitative assessments of efficiency can be used in the absence of the data required to calculate net present value, economic rate of return and estimates of the financial rate of return and include:
•	Actual costs compared with appraisal estimates and any revisions
•	Implementation delays and any redesign that may have increased costs
•	The level of benefits and their growth curve compared with expectations (if feasible)
•	Utilisation rates for project facilities and services
•	Services and facilities compared to good practice standards
•	Adequacy of the benefits stream compared with the costs.
* Qualitative assessments should always rely on an appreciation of the underlying concepts of cost-benefit analysis, the use of good practice in similar situations and other suitable indicators.

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Box 12.5:		Factors affecting sustainability
Factors to be considered when assessing the sustainability of impacts include:
•	Social support (including continued participation of beneficiaries and local communities, robustness of grass-roots organizations)
•	Technical soundness
•	Government commitment (including key central and local agencies and available operating funds)
•	Commitment of other stakeholders (including NGOs, local organizations, civil society and private sector)
•	Financial viability (including funding of rural organizations, role of cost recovery, capacity to finance recurrent cost, operationally and financial self-sufficiency and positive cash flows in marketing schemes)
•	Institutional support (including the legal/regulatory framework and organizations and management effectives)
•	Environmental impact and protection
•	Resilience to exogenous factors (such as price variability and market access, natural disasters and unstable security in the project area)
•	Replication (of a project approach as an indicator of sustainability).

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Box 12.6:		Key questions to assess gender equality and women's empowerment
•	Did the project design correctly identify gender-differentiated development opportunities?
•	Were adequate operational measures included in the design to realise this potential?
•	Has the implementation environment, including the institutional environment, been supportive of/conducive to such development?
•	Have women and men had equal opportunities to participate in overall project activities and in individual components?
•	Have women and men benefited equally from the overall project impact?
•	Has the project been innovative in creating gender-equal opportunities and in empowering women?
•	Has the project facilitated progress in gender-sensitive policies or government actions?
(See #8 - Addressing Gender in Development Projects for additional questions)

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Box 12.7:	Key evaluation questions to assess the performance of project partners:
Implementing/ primary organization (NB some of these questions may reflect IFAD's internal strategies)	•	Did the project design illustrate the connection with country and regional strategies?
	•	Was the design process participatory?
	•	Has the design adequately defined and addressed the project goals and objectives?
	•	Did the design adequately address targeting, participation and gender issues?
	•	Was the design suitable and modified if necessary?
	•	Did resource allocations accord with project objectives/goals?
	•	Are objectives currently relevant to target group's aspirations/needs?
	•	Did the organization support partners by taking prompt action when required?
	•	Has the project been innovative in design and implementation?
	•	Has the organization promoted the replication or scaling-up of the project components?
	•	Did the organization help to enforce the cooperating institution's recommendations?
	•	Was the project designed to be sustainable?
	•	Has the organization actively created an effective partnership for implementation?
	•	Has the organization sought to influence poverty policies?
Cooperating institutions (CI)	•	Has the supervision programme been well arranged (frequency, composition, continuity)?
	•	Has the CI followed compliance with loan covenants?
	•	Has the CI been effective in financial management?
	•	Has the CI sought to monitor project impacts and the concerns of the organization?
	•	Have implementation problems been highlighted and appropriate remedies suggested?
	•	Has the CI promoted or encouraged self-assessment and learning processes?
	•	Has the supervision process enhanced implementation and poverty impacts?
Government agencies	•	Has government correctly assumed ownership and responsibility for the project?
	•	By its actions and policies had government been fully supportive of project goals?
	•	Did government contribute adequately to the project preparation process?
	•	Have adequate staffing and project management been assured?
	•	Has technical assistance been contracted and used well when needed?
	•	Has appropriate staff training been arranged and funded?
	•	Have the appropriate levels of counterpart funds been provided on time?
	•	Have administrative arrangements (e.g. procurement) been suitable for implementation?
	•	Have participatory approaches included in the design been encouraged and/or adopted?
	•	Have loan covenants and the spirit of the loan agreement been followed?
	•	Has coordination of the implementing agencies been well arranged?
	•	Has policy guidance been provided to the project management where necessary?
	•	Has the project management discharged its functions adequately?
	•	Has government facilitated the work of NGOs and civil society where appropriate?
	•	Has adequate reporting and auditing been arranged?
	•	Has government planned an exit strategy and/or made arrangements for continued project activities?
NGOs and CBOs	•	Have NGOs/CBOc been involved in the project as envisaged?
	•	How effectively have NGOs fulfilled their contractual service agreements?
	•	Have the NGOs/CBOs acted to strengthen the capacity of rural poor organizations?
	•	Will NGOs/CBOs contribute to the sustainability of project activities?
Cofinanciers	•	Were cofinanciers well chosen in terms of congruence of mandates?
	•	Have adequate and timely resources been made available as agreed?
	•	Have administrative arrangements worked well?
	•	Have cofinanciers been active in encouraging project implementation?
	•	Has there been adequate coordination with cofinanciers?
	•	Is there potential for scaling up or continuing of cofinanciers' contributions actions?

Source: IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome

Further considerations

The matrices should be used as supporting tools and not as summary assessments - they assess ‘what’ has happened but do not address ‘why’ it has happened. They should inform the development of evaluation tools rather than being completed directly.
Only those boxes in each matrix that are relevant to individual projects should be completed.
Determining values for each cell in the matrices will require balancing the different perceptions of actors involved in the evaluation process.
Project intentions may need to be reconstructed if the original project objectives are too general to use in the evaluation or they have not been developed using a logical framework.
Stakeholder perceptions of changes that have occurred and the extent to which the project is responsible must be supported by other objective data and consultation with partners.
Evaluations should precede the development of an action-oriented document proposing how the recommendations will be implemented.

Background

The criteria proposed in this framework are based on the IFAD Strategic Framework for IFAD 2002–2005. This framework is based on IFAD’s Office of Evaluation’s experience in evaluating more than 200 IFAD projects.

While this framework was developed for specific use within IFAD, there is some flexibility in the framework which enables it to be applied to development projects in general. The tool has been reproduced here in general terms so that it can be applied across development projects in general. However, there has been no intent to change the meaning of the original material.

Other relevant frameworks, approaches and tools

# 11: The Auto-evaluation Method
# 13: Grassroots Development Framework
# 14: Impact Monitoring and Assessment
# 15: Selecting Indicators
# 16: Participatory Development of Indicators

References

IFAD, 2003. A methodological framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation, IFAD, Rome http://www.ifad.org/gbdocs/eb/ec/e/34/EC-2003-34-WP-3.pdf

13 - Grassroots Development Framework

(also called ‘the Cone’)
The Inter-American Foundation

A conceptual framework for planning, monitoring and evaluating development projects.

When can the Grassroots Development Framework help?

There is an over-emphasis on the tangible, measurable results of a project and the direct impacts on intended beneficiaries
Valuable, unanticipated impacts are not be taken into consideration
There is a long lag time before evaluation results can inform decision making
Evaluations are conducted too soon after project completion to measure long-term impacts
There is an over-emphasis on short-term results
Evaluations cannot be compared across different projects

What can the Grassroots Development Framework be used for?

Informing the development of organizational objectives, goals and directions
Comprehensive planning, monitoring and evaluation of development projects incorporating all areas of potential impacts
Identifying the expected impacts and areas of influence of a development project
Measuring the results and impacts of development projects
Comparing and contrasting the related projects across programmes, institutions, geographic areas and/or over time.

Why use this Grassroots Development Framework?

Evaluate the broader economic and social impact and the intangible impacts of development projects (as well as anticipated, tangible and direct results)
Link monitoring and evaluation to the planning process
Incorporate data collection methods that are culturally and organizationally appropriate
Focus on the process (means) of development as well a products (ends) to pursue long-term goals of poverty
Ensure that all areas of potential impacts and the interactions between them are considered in project planning and evaluation.

Key concepts

There are three levels of impacts in the Grassroots Development Framework: 1) Direct benefits (individuals or families); 2) Organisational impacts (NGOs, CBOs, networks); 3) Broader impacts on society (community, regional and national).

Tangible impacts: Are physical and material impacts that can be measured and substantiated by direct evidence.

Intangible impacts: can only be observed or inferred and are harder to measure in a quantitative manner.

Source: Pasteur, K. 2002. Analysing the Cone: A review of literature and experiences relating to the Grassroots Development Framework, IDS

Outline of the Grassroots Development Framework

The Grassroots Development Framework outlines six categories of impacts that should be assessed (see Box 14.1). The indicators within these categories are shown in Box 13.2.

Box 13.1: Six categories of impacts to be considered

* For development projects to be sustainable, they should address all the dimensions of the Grassroots Development Framework.

Source: Pasteur, K. 2002. Analysing the Cone: A review of literature and experiences relating to the Grassroots Development Framework, IDS

Box 13.2:	Grassroots Development Framework Indicators
Standard of Living:
1. Basic Needs	1.1	Number of beneficiaries who have improved or are improving their standard of living as a result of project activities
1. Basic Needs	1.2	Number of beneficiaries whose socioeconomic conditions changed owing to project activities themselves and not the economic or political situation in the country
2. Jobs and income	2.1	Number of beneficiaries who through the project, obtained new jobs, improved those they have (better pay and/or working conditions), or kept those that otherwise would have been eliminated
2. Jobs and income	2.2	Average income received by beneficiaries in the last 6 months from project activities
3. Assets	3.1	Total annual liquid and/or fixed assets that beneficiaries accumulated as a result of project activities
3. Assets	3.2	Number of beneficiary families according to the level of fixed and liquid assets they have as a result of grant activities.
Personal Capability - Knowledge, Skills and Attitudes (Personal Capacities in Box 13.1):
4. Knowledge and Skills	4.1	Number of beneficiaries who acquired knowledge and/or skills through courses, seminars or job training sponsored by the project
	4.2	Number of beneficiaries who applied the new knowledge and/or skills to their work as a result of project activities
	4.3	Number of project beneficiaries who increased their ability to lead or guide others in accomplishing project activities and goals
	4.4	Number of beneficiaries who improved their capacity to communicate their ideas and views clearly through project activities
5. Attitudes and values	5.1	Number of beneficiaries who report having the capacity, obtained through project activities, to act in their own benefit and to improve their standard of living
	5.2	Number of beneficiaries in terms of their appreciation of, care for, and presentation of their cultural values and traditional, and ethnic heritage as a result of project activities
	5.3	Number of beneficiaries that developed respect for and appreciation of other people's traditions, cultural customs and races as a result of project activities
	5.4	Number of beneficiaries with the capacity, obtained through project activities, to continue to devote time and energy to an activity, project, or goal until it is accomplished, or at least until it is determined that it is not feasible
	5.5	Number of beneficiaries who utilize and/or adapt more effective and/or efficient strategies, methods, or approaches to achieve project objectives.
Organizational Capacity and Culture (shown separately in Box 13.1):
6. Management	6.1	Demonstrated capacity of organization for planning, monitoring and evaluation of project activities
6. Management	6.2	The organization's demonstrated capacity to identify and use more effective strategies, methods and/or approaches to achieve project objectives
7. Implement-ation/ Administration	7.1	Number and average amount of loans given by the organization to its beneficiaries
	7.2	The organization's demonstrated capacity to efficiently manage financial, human or material resources and allocate the resource to priority needs and supervise their use to achieve institutional goals and/or project activities
	7.3	Value of the income generating organization's profits earned as a result of project activities
8. Resources	8.1	Total amount of resources mobilized (received) by the organization from other organizations that supported projects with same funding source
	8.2	The capability of the organization to acquire resources for its own use, from various external sources
	8.3	Monetary, material or human resources from national or international public or private entities that the grantee obtained and channeled directly to other grassroots organizations or groups that support the project. The resources brokered never pass through the organization's hands but rather to directly to other organizations, grassroots groups, or even to beneficiaries themselves
9. Vision	9.1	Capacity demonstrated by the organization to establish and modify long-term goals and plans of action, beyond the goals of the current project and enabling the project to be sustained after project funding has ended
9. Vision	9.2	The organization's demonstrated capacity to foresee economic, political or market conditions and to react appropriately to the situation
10. Participatory practice	10.1	Demonstrated willingness of the organization to provide information to its staff, beneficiaries, and other organizations involved, if any, on its policies, programmes and finances
10. Participatory practice	10.2	The organization's demonstrated practice of consulting its staff, partners, and/or beneficiaries on decisions affecting project goals and operations
11. Organizational relationships	11.1	Number and types of other organizations that have established informal and/or legal partnerships with the organization. Partner organizations agree to work jointly to fulfill the project's goals and objectives. A partner organization is one contributing financial, human or material resources to support the project's objectives and participating in the decision making process
11. Organizational relationships	11.2	The number and type of other organizations that have established formal and/or legal partnerships with the organization. Partner organizations contribute financial, human or material resources to support the project's objectives and participate in the decision making process
Policy Environment (Laws/Policies):
12. Laws	12.1	Number of laws, statutes, regulations and other legal provisions which the organization helped to enact as a direct result of project activities
12. Laws	12.2	The number of laws, statutes, regulations and other legal provisions that were implemented as result of the activities of the organization
13. Policies	13.1	Number of topics discussed by the organization at meetings or in the media as a result of grant activities. Such discussions promote civil society, the organization and cooperating organizations
13. Policies	13.2	Number of policies or plans of action designed and implemented by the organization as a result of project activities
14. Dissemination and replication	14.1	Number of speeches or presentations made or products developed for purposes of disseminating project approaches, practices or techniques. Such presentations and/or products are directed at beneficiaries or other interested parties
14. Dissemination and replication	14.2	Number of individuals and organizations, excluding the organization, which adopted the approaches, methods, or techniques proposed or adopted in the project
Community Norms (attitudes and values):
15. Awareness	15.1	Demonstrated capacity of organization to raise public awareness regarding the disadvantaged population, that is benefiting from the project
16. Practices	16.1	The demonstrated capacity of the organization, as a result of the grant, to influence the government and/or state institutions to accord more favourable treatment to disadvantaged people
17. Relations	17.1	Influence of the organization in maintaining productive working relations with civil society entities or organizations, other than its relations with partner organizations
17. Relations	17.2	Influence of the organization on public sector entities or organizations other than its partner government agencies (local, regional, national).

Source: Inter-American Foundation, (no date). Grassroots Development Framework, http://www.iaf.gov/grants/grassroots_dev_framework_text_en.asp?grass=1

Further considerations

The six categories of impacts are not independent of each other and are inter-related.
Estimation and opinion are the main forms of measurement but quantitative data is also used.
A participatory approach should be used throughout project design and formulation, implementation and evaluation and not only to extract data but to broaden the scope of evaluation and empower stakeholders.
In order to compare evaluations, a standardized instrument must be used to collect data. This requires a trade-off in the flexibility in the use of data collection methods tailored to specific project contexts.
The Grassroots Development Framework can be adapted to a diversity of different contexts and is not biased towards any particular sector. It has been used in agriculture, environment, health, gender and community development sectors.
Additional variables can be included in the Grassroots Development Framework to tailor it to specific projects. However, the inclusion of too many variables may overcomplicate the framework.
Criteria have been developed to assess the capacity of an organization that is being funded by a donor agency to deliver a development project but may be applied in a broader context.
The Grassroots Development Framework is still considered to be in development and it is still being built on and adapted in response to lessons learned from its use.

Background

This tool was developed from the experience of monitoring 4,000 projects financed by the Inter-American Foundation. While it was developed for use within the Inter-American Foundation, it has also been adopted by a number of Latin American organizations for their own internal use. Therefore it is included in the SARD Project Toolkit for use in development organizations generally.

Other relevant frameworks, approaches and tools

# 15: Selecting Indicators

References

Inter-American Foundation, (no date). Grassroots Development Framework, http://www.iaf.gov/grants/grassroots_dev_framework_text_en.asp?grass=1

Pasteur, K. 2002. Analysing the Cone: A review of literature and experiences relating to the Grassroots Development Framework, IDS, http://www.livelihoods.org/info/tools/Thecone.pdf

14 - Impact Monitoring and Assessment

Swiss Agency for Development and Cooperation (SDC) and
Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ)

A participatory framework for incorporating impact assessment into the project cycle.

When can Impact Monitoring and Assessment help?

Monitoring project performance focuses on inputs, outputs and outcomes but does not take changes in the project context into consideration
Project goals are not kept in sight during project implementation
There is an over-emphasis on assessing project efficiency compared to project effectiveness
Attributing impacts or changes in the project context to project actions is difficult.

What can Impact Monitoring and Assessment be used for?

Observing and interpreting the changes in the project context
Adapting and improving projects in response to changes in the project context
Identifying plausible relations between actions and changes in the project context
Identifying project impacts and indicators to measure project impacts
Involving and empowering stakeholders
Building stakeholder capacity to present views, analyze, negotiate and make decisions
Determining the extent to which a development project has achieved its purpose/goal.

Why use Impact Monitoring and Assessment?

Broaden the focus of project monitoring to include the project context
Integrate impact monitoring assessment into the project cycle
Conduct impact assessment as part of project self-evaluation
Ensure continued appropriateness and effectiveness of project activities.

Key concepts

Context: The biological, socio-cultural, economic, institutional and political milieu or environment in which a development project is conducted.

Sustainable Land Management (SLM): the use of renewable land resources (soils, water, plants and animal) for the production of goods to meet changing human needs while protecting the long-term productive, physiological, cultural and ecological functions of those natural resources for the benefit of society. Land management becomes more sustainable if progress is made in the social/institutional, economic and ecological dimensions at the same time.

Source: Herweg, K. and Steiner, K., 2002. Impact Monitoring and Assessment: Instruments for Use in Rural Development Projects with a Focus on Sustainable Land Management, Volume 1, Centre for Development and Environment, Switzerland and Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ), Germany

Outline of the Impact Monitoring and Assessment Framework

Table 14.1 outlines the Impact Monitoring and Assessment Framework.

Table 14.1: Impact Monitoring and Assessment in the Project Cycle

Project Cycle	Steps and Key Questions for Impact Monitoring and Assessment (IMA)
1.PlanningStakeholder analysis	Step 1: Involvement of stakeholders and information management Stakeholders must be involved in stages 2 to 6 to provide their knowledge and perceptions of the project context and to balance the views, values and objectives of different groups and manage conflicts. Ensure that information is accessible for all stakeholders.
	Key questions:
	•	Who should participate in IMA?
	•	Who can provide and who needs what information, and in what form?
	•	How will information be disseminated and stored so it is accessible for anyone?
Problem analysis	Step 2: Review of Problem Analysis Identify starting points for project activities.
	Key questions:
	•	What are the most important elements of the project context and how are they interlinked?
	•	What role do they play in the context?
	•	Is the context moving towards or away from sustainability?
Objectives analysis (Analysis of alternative solutions)	Step 3: Formulation of Impact HypothesesIdentify how the project can intervene, which elements/links of the system it will affect, the situation to be achieved (goals/purpose) and the expected resultsDefine the impact hypothesis (the use of project outputs, the resultant effect, benefits/drawbacks arising and subsequent impacts)
	Key questions:
	•	What contribution can the project make towards more sustainable development?
	•	What positive and negative impacts might arise?
Indicator selection	Step 4: Selection of impact indicatorsTo determine if any change in the context is contributing to the project purpose/goal and communicate this to stakeholders and partners.
	Key questions:
	•	What indicates a change in the project context?
	•	What reveals which impact hypotheses materializes?
	•	What set of indicators will tell if changes contribute to the project purpose and goal?
	•	Can local indicators be used (see # 16)?
	•	How can a reasonable number of indicators be selected (see # 15)?
	•	How can impact assessment be prepared?
2. Monitoring	Step 5: Development and application of impact monitoring methods
	Key questions:
	•	How can the context and impact indicators be monitored and documented?
	•	Which methods are applicable within the means and capacities of the project (e.g. cost-effective, flexible and able to be used by staff) but still meet quality requirements?
	•	How can methods be best combined for triangulation?
3. Evaluation	Step 6: Impact assessmentFind relations between project outputs and changes in the context.
	Key questions:
	•	How did the context change in the eyes of different stakeholders (at the household level, at community level, at other levels)?
	•	What did they learn from these changes?
	•	Do the lessons learned indicate that the project has stimulated important social processes (e.g. individuation, self-determination, empowerment, innovation, adaptation, ethnic integration, participation, social learning etc)?
	•	What plausible relationships can be identified between the project, social processes and changes in the context? Would the changes have occurred anyway i.e. even without the project? Which factors - alone or in combination - have contributed to the changes?
	•	Do changes in these social processes contribute to (development) goals?
	•	Which processes should be strengthened specifically in the future?

Source: Herweg, K. and Steiner, K., 2002. Impact Monitoring and Assessment: Instruments for Use in Rural Development Projects with a Focus on Sustainable Land Management, Volume 1, Centre for Development and Environment Switzerland and GTZ, Germany

Box 14.3:		Questions to address in the follow up to IMA
•	Are there new stakeholder groups that should be involved during the next project phase? (Step 1)
•	Is the analysis of the project context still relevant and representative? (Step 2)
•	Do the impact hypotheses have to be revised or supplemented, after initial changes and impacts appear? (Step 3)
•	Is the selection of impact indicators still relevant, and can it represent all important changes? (Step 4)
•	Did the monitoring methods applied produce useful data and information? How can methods be optimized or simplified? What should be added or omitted? (Step 5)
•	Was the impact assessment satisfactory or does it need to be modified? (Step 6)

Source: Herweg, K. and Steiner, K., 2002. Impact Monitoring and Assessment: Instruments for Use in Rural Development Projects with a Focus on Sustainable Land Management, Volume 1, Centre for Development and Environment Switzerland and GTZ, Germany

Further considerations

Impact Monitoring and Assessment (IMA) is only one option for determining the impact of development projects.
IMA can be adapted to suit project-specific conditions and local contexts.
Although developed for sustainable land management projects, the generic method can be applied to projects “in other sectors such as health, education and infrastructure”.
IMA is designed to be carried out by project staff. It employs, simple, cost-effective tools and instruments that have been tested in practice.
IMA should be complemented by performance assessments that measure achievement in terms of inputs, outputs and outcomes.

Background

This tool was initially developed by the Swiss Agency for Development and Cooperation (SDC) and the GTZ (Deutsche Gesellschaft fur Technische Zusammenarbeit) in conjunction with Intercooperation and Helvetas (Switzerland). Development began in 1996/97 and the version summarised here is based on feedback provided by experts who tested the framework in the field between 1998 and 2001. It builds on the Guidelines for Impact Monitoring for Sustainable Land Management released as working documents in 1998.

Other relevant frameworks, approaches and tools

# 13: Grassroots Development Framework
# 15: Selecting Indicators

References

Herweg, K. and Steiner, K., 2002. Impact Monitoring and Assessment: Instruments for Use in Rural Development Projects with a Focus on Sustainable Land Management, Volume 1, Centre for Dvelopment and Environment, Switzerland and Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ), Germany

http://www.tropentag.de/2002/proceedings/node84.html#1597%20

15 - Selecting Indicators

Reed, M. and Dougill, A. and The World Bank

A checklist of criteria for selecting indicators for monitoring the inputs, outputs, outcomes and impacts of development projects.

When can these criteria for selecting indicators help?

Project outcomes and impacts cannot be demonstrated
Project outcomes and impacts cannot be clearly linked to project actions
Accountability for project delivery cannot be traced
Project benefits cannot be ascribed to different social groups
Monitoring programmes cannot be implemented with existing resources and capacity and within decision making time frames.

What can these criteria be used for?

Measuring progress in project implementation and adapting project activities as necessary
Measuring project outcomes and impacts
Reflecting the causal links between inputs, outputs, outcomes and impacts
Tracking accountability for project implementation and delivery.

Why use these criteria?

Apply a consistent process to all project scenarios and across all sectors
Accommodate the complexity of sustainability in indicators for evaluation
Inform project management and decision making
Provide an objective basis to the process of selecting indicators.

Key concepts

Indicators: Variables used to measure progress towards goals or objectives to be achieved, including measures of inputs, outputs, outcomes and impacts.

Source: The World Bank, 2004. Selecting Indicators, Poverty Monitoring Guidance Note 1, The World Bank, Washington D.C.

Criteria for selecting indicators

The selection of indicators should be guided by the criteria outlined in Box 15.1.

Box 15.1:		Evaluation criteria for selecting sustainability indicators
Indicators should:
•	Be easily measured
•	Be rapid to measure
•	Be sensitive to spatio-temporal change (and variable across social groups)
•	Assess trends over time and provide early warning of detrimental change, progress and improvement
•	Be cost-effective to measure (and able to be measured within existing capacity and resources)
•	Be easy to understand and interpret
•	Be reliable (not dependent on the interpretation of the user) and robust (difficult to manipulate or be blown off course by unrelated developments and not sensitive to external or exogenous factors (e.g. require self-reporting or are linked to incentives that may encourage over-or under-estimation)
•	Be representative of system variability and applicable over different regions
•	Be timely
•	Be scientifically credible
•	Be verifiable and replicable
•	Be consistent over time
•	Have social appeal and resonance
•	Be policy relevant (in terms of level of aggregation and the timing of the decision-making cycle)
•	Make use of available data
•	Be locally relevant
•	Be accurate
•	Be free from bias
•	Be derived by the users
•	Simplify complex phenomena
•	Quantify information so that its significance is readily apparent
•	Facilitate communication of information, particularly between data collectors and users
•	Able to be disaggregated according to factors relevant to project activities, goals and decision (e.g. geographic areas, gender, age groups, income, consumption or asset ownership levels and ethnic, religious, tribal and other social groups).

Sources:
Reed, M.S. and Dougill, A.J., 2003. Facilitating Grass-Roots Sustainable Development through Sustainabilit y
Indicators: A Kalahari Case Study, presentation to “Frontiers 2: European Applications in Ecological
Economics”, 12–15 February 2004, Canary Islands
The World Bank, 2004. Selecting Indicators, Poverty Monitoring Guidance Note 1, The World Bank,
Washington D.C. (criteria from this source are denoted in brackets)

Further considerations

The World Bank recommends selecting few indicators “to answer the right questions at the right level of disaggregation” and that are “of good quality and easily measurable” within existing capacity and available resources”.
Selecting indicators to a scientific and political process and should be done in consultation with communities and decision makers.
Qualitative and quantitative indicators are complementary and provide different types of information. Qualitative indicators provide important descriptive information.
The quality of an indicator can only be considered in the context of its intended use.
Measurement of performance indicators (measures of inputs, outputs, outcomes and impacts) should be accompanied by the measurement of overall performance and the context in which the project is being implemented.

Background

This checklist is a combination of two checklists, one developed by the World Bank for monitoring poverty reduction projects and the other developed from a review of 22 publications by Reed and Dougill in their development of a methodology for developing sustainability indicators.

Other relevant frameworks, approaches and tools

# 11: The Auto-evaluation Method
# 12: Methodological Framework for Project Evaluation
# 13: Grassroots Development Framework
# 14: Impact Monitoring and Assessment
# 16: Participatory Development of Indicators

References

Reed, M.S. and Dougill, A.J., 2003. Facilitating Grass-Roots Sustainable Development through Sustainability Indicators: A Kalahari Case Study, presentation to “Frontiers 2: European Applications in Ecological Economics”, 12–15 February 2004, Canary Islands

The World Bank, 2004. Selecting Indicators, Poverty Monitoring Guidance Note 1, The World Bank, Washington D.C. http://siteresources.worldbank.org/INTPAME/Resources/Selective-Evaluations/NoteIndicators_eng_Apr04_doc.pdf

16 - Participatory Development of Indicators

M.S. Reed and A.J. Dougill

A framework for developing sustainability indicators that can be used by land users to guide sustainable land management.

When can the participatory development of indicators help?

Frameworks developed to classify or guide the development of indicators have been developed from a theoretical perspective
Sustainability indicators are often designed for use at national and international scales and have little meaning for land users
Land users require specialist training and equipment to use sustainability indicators
Indicators do not usefully inform land management decisions.

What can the Participatory Development of Indicators be used for?

Identifying indicators to assess sustainability
Accounting for social factors influencing indicator use and the development of indicators
Identifying new indicators relevant to land users and local environments
Broadening the scope of sustainability indicators identified for development projects
Integrating local and scientific knowledge for indicators that are accurate and easy to use
Developing more meaningful interpretations of existing sustainability indicators
Developing indicators that can be used by land managers without training or specialist equipment
Identifying indicators that can feed practically into sustainable land use management decisions.

Why use the Participatory Development of Indicators?

Focus research at local scales relevant to land users
Facilitate practical improvements in sustainable land use management practices
Link sustainability indicators to land management objectives
Reduce trade-offs between meaningful participation and scientific rigor
Encourage locally targeted responses to degradation and sustainability
Improve transparency in decision making through participation.

Key concepts

Indicators: variables used to measure progress towards goals or objectives to be achieved, including measures of inputs, outputs, outcomes and impacts.^A

Sustainability Indicators: indicators that look at the economic, social and environmental information in an integrated manner ^B

Sources:
^A The World Bank, 2004. Selecting Indicators, Poverty Monitoring Guidance Note 1, The World Bank, Washington D.C.
^BTschirley, J., 1996. Use of Indicators in Sustainable Agriculture and Rural Development, Sustainable Development, FAO, Rome

The participatory framework for developing sustainability indicators


*Notes on the process for developing sustainability indicators:*
1.	Participation in identifying the objectives for indicators and criteria to select indicators is important because a range of criteria can be perceived by different land users. This can be done using semi-structured interviews and village-level focus groups.
2.	A livelihoods approach is taken to guide research into the multiple dimensions of rural livelihoods, to explain differences in knowledge about indicators and ensure indicators meet the needs of all sectors of the community. This can be done using semi-structured interviews, oral histories, rangeland walks and time-line discussions.
3.	Land managers identify signs they use to determine changes or describe success and failure in relation to their farming systems.
4.	Indicators identified by land users are combined with sustainability indicators from the literature that have been developed in comparable environments.
5.	The combined indicator list is evaluated according to accuracy, relevance and easy of use using village-level focus groups.
6.	The reliability and applicability of community-identified indicators are determined.
7.	The assessment of indicators is discussed in interviews and targeted focus groups.
8.	(to10) Management strategies to achieve improvement in the selected indicators are

Source: Reed, M.S. and Dougill, A.J., 2003. Facilitating Grass-Roots Sustainable Development through Sustainability Indicators: a Kalahari Case Study, Presentation at “Frontiers 2: European Applications in Ecological Economics” , 12–15 February 2003, Tenerife, Canary Islands, Leeds Environment and Development Group, School of the Environment, University of Leeds, Leeds, UK

Further considerations

The need for an agreed working definition of sustainability is a challenge when engaging communities in the identification of sustainability indicators.
The authors advise that more work is required to “validate” and “optimise” this approach.
Different social groups will use locally-identified indicators differently and this has implications for aggregating local data to inform decisions at larger scales and making comparisons between local areas. However, this trade-off may be justified in order to drive action at the local level.
Community-identified indicators may include surrogates that are qualitative but accurate enough to support management decisions.
Community-identified indicators should be complemented by indicators chosen to monitor progress towards sustainability at national and international scales.

Background

This framework was developed by the Leeds Environment and Development Group and was tested with land users in the south Kagalagadi district, Botswana.

Other relevant frameworks, approaches and tools

# 15: Selecting Indicators

References

Reed, M.S. and Dougill, A.J., 2003. Facilitating Grass-Roots Sustainable Development through Sustainability Indicators: a Kalahari Case Study, Presentation at “Frontiers 2: European Applications in Ecological Economics” , 12–15 February 2003, Tenerife, Canary Islands, Leeds Environment and Development Group, School of the Environment, University of Leeds, Leeds, UK

The World Bank, 2004. Selecting Indicators, Poverty Monitoring Guidance Note 1, The World Bank, Washington D.C.

Tschirley, J., 1996. Use of Indicators in Sustainable Agriculture and Rural Development, Sustainable Development, FAO, Rome, http://www.fao.org/sd/EPdirect/EPan0001.htm