Sustainable Development Banner

Posted November 2000

Monitoring and evaluating stakeholder participation in agriculture and rural development projects: a literature review

By Marilee Karl


Monitoring and evaluating stakeholder participation in agriculture and rural development projects: an annotated bibliography

Summary

Development agencies began to introduce concepts of participation in projects and programmes in the late 1970s and early 1980s after lack of beneficiary participation was identified as a reason for the failure of many development efforts. Initially, emphasis was on popular participation. In the past decade the promotion of participation in development has become more widespread and the focus has widened to include other stakeholders as well.

Participation can take place in different places of the project cycle and at different levels of society, and take many different forms. These can range along a continuum from contribution of inputs to predetermined projects and programmes, to information sharing, consultation, decision-making, partnership and empowerment. Participation is both a means and an end. As a means, it is a process in which people and communities cooperate and collaborate in development projects and programmes. As an end, participation is a process that empowers people and communities through acquiring skills, knowledge and experience, leading to greater self-reliance and self-management.

Development agencies have only recently recognized the need for monitoring and evaluation (M&E) of participation in development projects and programmes. Three aspects of participation need to be evaluated:

  1. the extent and quality of participation;
  2. the costs and benefits of participation to the different stakeholders; and
  3. the impact of participation on outcomes, performance and sustainability.

Because participation is a qualitative process, conventional M&E methodologies are inadequate to evaluate participation.

Some of the common objectives and expected benefits of participation in development are improving the efficiency, effectiveness, sustainability and coverage of projects and programmes and promoting stakeholder capacity, self-reliance and empowerment. Participation also entails costs to the different stakeholders. There are time costs for both the beneficiaries and the donors, including increased time for training, preparation and consultation. There are likely to be financial costs to the donors. A hidden cost may be the shifting of responsibilities from national governments to the rural poor.

In order to monitor and evaluate stakeholder participation in development projects and programmes, it is necessary to identify the stakeholders, i.e. those who are affected by the outcome, negatively or positively, or those who can affect the outcomes of a proposed intervention. Primary stakeholders are those people and groups who are ultimately affected by the project. Secondary stakeholders are intermediaries in the process of delivering aid to primary stakeholders. External stakeholders are those not formally involved in a project, but who may impact or be impacted by it. In development projects and programmes, stakeholders usually include donor agencies, government, civil society organizations and the local community and beneficiaries. Stakeholder analysis is one of the major methods used for identifying the relevant stakeholders of a particular project or programme.

Experiences in monitoring and evaluation of participation are still limited. Attention has been focused more on identifying stakeholders and assessing the extent and quality of stakeholder participation than on assessing the costs and benefits of participation to the different stakeholder groups or the impact of stakeholder participation. However, there is not always a clear separation among the approaches and methods for assessing these different aspects of participation.

Assessing the extent and quality of participation requires both quantitative and qualitative indicators. Quantifiable indicators can be used to measure the economic aspects of participation, the extent of participation in organizations and project activities, and the development momentum. Qualitative indicators measure processes such as organizational growth, group behaviour and self-reliance. These indicators may evolve over the life of a project as participation changes. Monitoring is a crucial element in assessing the extent and quality of participation, as is interpretation of the qualitative indicators. Understanding gender differences is also essential. While different methods can be used, participatory monitoring and evaluation and involvement of the primary stakeholders wherever possible is generally recommended.

Experiences in assessing the costs and benefits of participation to the different stakeholders have been limited. The few assessments documented in the literature pertain mainly to the costs to the donor agency. The World Bank, in particular, has carried out some assessments of the costs of participation to the Bank. Results show that participatory projects require more financial and time input than non-participatory projects, but that these are not excessive. The studies have not, however, attempted to measure the costs of not providing for participation. Some of the literature suggests that participatory monitoring and evaluation could be used to assess the costs and benefits of participation to the primary stakeholders.

Assessments of the impact of stakeholder participation have been carried out mainly through reviews of ex post evaluations, case studies, surveys and statistical analysis. There has also been some experience of using both conventional M&E techniques and participatory M&E during the life of the projects. While evidence is still limited, it suggests that participation has a positive impact on project and programme performance, outcomes and sustainability.

Introduction

Although the promotion of participation and participatory processes in development projects and programmes is not new, evaluation of participation is still in its infancy. Multilateral and bilateral development agencies have only recently begun to recognise the need for evaluating participation (Oakley 1988, 1992: Rudqvist and Woodford-Berger 1996, Uphoff 1992).

This literature review attempts to document the work that has been done on evaluating participation in development projects and programmes. It covers materials that deal, directly or indirectly, with approaches to evaluating the extent and quality of participation, the costs and benefits of participation to stakeholders and the impact of stakeholder participation on the performance, outcomes and sustainability of projects and programmes.

These have been winnowed out of a much larger mass of materials dealing with evaluation and participation in general and with participatory monitoring and evaluation. Because evaluations explicitly addressing participation are only now beginning to emerge, the amount of materials directly addressing evaluation of participation, rather than participation in evaluation, is still limited.

Some of the materials on participatory monitoring and evaluation, however, do address issues of evaluation of participation. Information relevant to the evaluation of participation is also found in materials dealing with stakeholder analysis, conflict management and impact analysis.

Although the focus of this literature review is on evaluation of participation in rural development projects and programmes, some material is also included on participation in research and policy, as some of the lessons learned in these areas are also useful for projects and programmes.

This literature review draws on materials from a number of sources, including:

The Informal Working Group on Participation (IWG-PA) web site's library data base; the FAO library; bibliographies compiled by IWG / PA members; and the web sites of the Department for International Development (DFID), the Electronic Development and Environment Information System (ELDIS), the Inter-American Development Bank (IDB), the Institute for Development Studies (IDS), the International Development Research Centre (IDRC), the International Labour Organisation (ILO), the International Institute for Environment and Development (IIED), the Organisation for Economic Co-operation and Development (OECD), the Swedish International Development Authority (SIDA), the United Nations Development Programme (UNDP), and the World Bank.

Section II of this literature review focuses on stakeholder participation, including: concepts and definitions of participation; forms of stakeholder participation at different stages in the project cycle; the need, purposes and difficulties of monitoring and evaluating participation.

Section III examines the objectives of participation and hypotheses of its costs and benefits.

Section IV reviews approaches to monitoring and evaluating participation. It looks at: who are the stakeholders; how to measure the extent and quality of stakeholder participation; methods for evaluating the costs and benefits of participation to each stakeholder group; and methods for assessing the impact of stakeholder participation on project and programme performance, outcomes and sustainability.

Stakeholder participation

DEFINITIONS AND CONCEPTS OF PARTICIPATION

Definitions and concepts of participation in development have evolved over time. Their roots can be traced back to community and popular participation, promoted mainly by non-governmental organizations (NGOs) in the 1950s and 1960s. In the late 1970s and early 1980s, multilateral agencies, such as FAO, ILO and UNRISD, also began to promote popular participation in development projects and programmes (Rudqvist and Woodford-Berger 1996).

The failure of many conventional development projects and programmes and growing poverty brought a shift away from the modernisation paradigm of development in the 1970s. People were identified as the missing element in development efforts. The limited success of many development initiatives was attributed to failure to involve people in the design and implementation of projects and programmes (Cernea 1991, FAO 1990, Hinchcliffe et al 1995, Kottak 1991, Oakley 1991, Oakley and Marsden 1984, Uphoff 1991, World Bank 1998).

The main emphasis in the 1980s was, therefore, on popular or people's participation and on ways to involve the rural poor in development projects and programmes. One of the early initiatives to involve people in development was FAO's People's Participation Programme (PPP) with a focus on the rural poor, women in development and promotion of small groups in development projects and programmes (FAO 1990). This programme viewed participation as enabling the rural poor to pool their efforts and resources in pursuit of objectives set by themselves (FAO 1990).

UNRISD's Popular Participation Programme in the early 1980s defined participation as "the organised effort to increase control over resources and regulative institutions in a given social situation on the part of groups or movements hitherto excluded from such control" (cited in Rudqvist and Woodford-Berger 1996, p. 11).

There has been a growing emphasis on empowerment of people, a concept that has been widely promoted by NGOs (Oakley and Marsden 1984, Rudqvist and Woodford-Berger 1996). The concept of participation as empowerment has many nuances. Some development agencies see it basically as access to and control over resources, or as a way of releasing human energies and enlarging talents and potential (FAO 1990, Uphoff 1992). It has also been linked to democratisation, good governance, equality, equity, and human rights (FAO 1990, Rudqvist and Woodford-Berger 1996).

Popular participation can be interpreted along three broad lines (Oakley 1991):

Participation in development projects and programmes is widely seen as both a means and an end. While many development agencies give equal weight to both, some emphasize one or the other aspect of participation (Rudqvist and Woodford-Berger 1996).

As a means, participation is a process in which people and communities cooperate and collaborate in development projects and programmes (IDB, Clayton et al 1998). In this view, participation, sponsored by an external agency, is a way to support the progress of a project or programme and a means to ensure the successful outcome of activities. The term "participatory development" is commonly used to describe this approach (Clayton, et al 1998). Participation is also viewed as a means to help ensure sustainable development (Rudqvist and Woodford-Berger 1996, Uphoff 1992).

As an end, participation is seen as the empowerment of individuals and communities in terms of acquiring skills, knowledge and experience, leading to greater self-reliance (IDB, Clayton et al 1998). Participation is an instrument to break poor people's exclusion and lack of access to and control over resources needed to sustain and improve their lives. It is intended to empower them to take more control over their lives (Clayton et al 1998).

Concepts of participation have widened to include not only the rural poor but also other sectors of civil society. This is reflected in a change of terminology from "the rural poor", "beneficiaries" or "users" to "stakeholders" and "partners" (Rudqvist and Woodford-Berger 1996, World Bank 1998). The World Bank's Learning Group on Participatory Development defines participation as "a process through which stakeholders influence and share control over development initiatives and the decisions and resources which affect them" (World Bank 1996, p. xi).

Several factors have influenced this notion of participation as involving a wider range of stakeholders. One is the trend towards decentralisation and transfer to responsibilities from government to people. Another is the conclusion that small-scale community participation and empowerment are not sufficient to ensure the sustainability of development efforts (Rudqvist and Woodford-Berger 1996, Warren 1998).

With this has come an emphasis on partnership and dialogue among the various stakeholders (Rudqvist and Woodford-Berger 1996). In complex socio-political environments, the concept of participation has increasingly come to include "involvement of local institutions and civil society in a power-sharing scheme, based on negotiation and conflict management" (Warren 1998, p. 122).

FORMS OF STAKEHOLDER PARTICIPATION

Participation can take many forms. Development agencies and authors distinguish different dimensions, places, degrees and levels of participation in development projects and programmes (Rudqvist and Woodford-Berger 1996). These can refer to:

Four broad types of participation can be distinguished in popular participation projects and programmes (Oakley 1988):

Many development agencies and authors distinguish a continuum of participation, ranging from minimal participation to intense participation. The following table outlines several different views of this continuum.

Development projects and programmes.
(Clayton et al 1998)
World Bank-supported projects.
(World Bank 1996)
Participatory research.
(McAllister 1999)
World Bank Poverty Reduction Strategy. (Edgerton et al 2000, McGee and Norton 2000)
Manipulation: participation is contrived as the opportunity to indoctrinate.   Cooption: token participation, with no real input or power. Compliance: community is assigned tasks.  
Information: stakeholders are informed about their rights, responsibilities and options. Information sharing: projects designers and managers share information with beneficiaries.   Information sharing: one way flows of information to the public.
Consultation: stakeholders may express suggestions and concerns but have no assurance that their input will be used. Consultation: people are given the opportunity to interact and provide feedback to the development agency. Consultation: local opinion is sought, but analysis and decisions are made by outsiders. Consultation: two-way flow of information between the coordinators of the consultation and the public.
Consensus-building: stakeholders negotiate positions.   Cooperation: local people help determine priorities, but the process is directed by outsiders.  
Decision-making: stakeholders take collective decisions. Decision-making: people have a role in making decisions on policy, project design and implementation.   Collaboration: shared control over decision-making.
Risk sharing: stakeholders make decisions together and share in the risks.      
Partnership: stakeholders work together as equals towards mutual goals.   Co-learning: local people and outsiders share knowledge and work together to form action plans.  
Self-management: stakeholders interact in a learning process which optimises the well-being of all concerned. Initiating action: people are able to take the initiative in terms of actions and decisions pertaining to operations. Collective action: people set their own agenda and carry it out in the absence of external initiators. Empowerment: transfer of control over decision-making and resources to all stakeholders.

Another way of looking at forms of participation of the rural poor is along a continuum of empowerment (World Bank 1996). In this categorisation, the rural poor can be:

Looked at from the perspective of any of the stakeholders in a project or programme, participation can include a range of possibilities (DFID 1995c):

THE NEED, PURPOSES AND DIFFICULTIES OF MONITORING AND EVALUATING PARTICIPATION

Need

Recognition of the need for evaluation of participation in development projects and programmes is a recent phenomenon. This is because:

  1. participation has only recently gained widespread acceptance in many agencies; and
  2. there is still a gap between participation rhetoric in policies and participation as practice at the operational level. (Oakley 91, Rudqvist and Woodford-Berger 1996).

However, there is a growing recognition that if participation in one form or another is an objective of development projects and programmes, it must be evaluated (Bhatnagar and Williams 1992, Clayton et al 1998, Oakley 91, DFID 1995b, FAO 1980, IDB). The development of quantitative and qualitative techniques for assessing the costs, benefits, and long-term effects of participatory projects is, therefore, a pressing task (Rudqvist 1992).

Three main aspects of participation in rural development projects and programmes need to be evaluated:

There is also a broad agreement in the literature on the need for systematic procedures for monitoring and evaluating the process of participatory monitoring and evaluation (PM&E). While PM&E offers many potential benefits to project or programme success, it could also result in a waste of time and resources and the failure to notice problems, if it is carried out poorly or inappropriately (Estrella and Gaventa). There is a danger that participatory monitoring and evaluation may become a panacea for all the problems of monitoring and evaluation in agricultural projects. Therefore, it is crucial to determine whether participation contributes significantly to more effective monitoring and participation (Abbot and Guijt 1998).

Purposes

The purposes of monitoring and evaluation are closely linked to the perceived needs to monitor and evaluate participation, as well as to the objectives of participation in a particular project and programme. The following table summarizes the purposes of monitoring and evaluating participation in projects and programmes, the purposes of participatory monitoring and evaluation, and the purposes of monitoring and evaluating participatory research.

Development projects and programmes.
(Bhatnagar and Williams 1992, Huizer 1983, IDS 1998, Marsden et al 1994, Oakley 1998, Uphoff 1989, Uphoff 1992)
Participatory monitoring and evaluation (PM&E).
(Estrella and Gaventa, Abbot and Guijt 1998)
Participatory research for community-based natural resource management projects.
(McAllister 1999)
Impact assessment: to assess to impact of participation on project and programme outcomes and performance Impact assessment: to compare objectives and achievement and to determine whether project interventions are achieving the identified objectives; programme objectives remain relevant over time; and the best action strategies are being pursued. Conceptual learning: to improve understanding of how different participatory research approaches and methods influence the outcomes of natural resource management projects.
Accountability: to assess and justify the costs and other investments in participatory development. Public accountability: to hold project beneficiaries / programme recipients, donors and governmental institutions accountable to agreed goals and performance targets. Accountability: to justify the research strategy and expenses to funding agencies through illustrating the link between participatory research methods and project outcomes.
Project management and planning: to learn from experience and use it in replanning and designing, implementing, monitoring and evaluating new projects. Project management and planning: to gain information which can be used for improving project planning and implementation and to plan for future goals and activities. Project management: to systematically learn from and adapt the research approach as the project proceeds.
Stakeholder capacity-building: to learn from the extent, quality and process of stakeholder participation and to use the lessons learnt in order to strengthen the capacity of participants. Organizational strengthening or institutional learning: to enhance the sustainability, replicability, and effectiveness of development efforts through the strengthening of people's organizational capacities.  
  Understanding and negotiating stakeholder perspectives: to allow different stakeholders to articulate and present their needs, interests and expectations.  

Difficulties

There are a number of difficulties inherent in the monitoring and evaluation of participation. Throughout the literature, there is agreement that conventional M&E methodologies are inadequate to evaluate participation. The evaluation of participation, therefore, requires methodological adjustments. Participation is a qualitative process that cannot be measured using only quantifiable indicators. There is widespread agreement that it is necessary to develop qualitative indicators in order to evaluate participation (Bhatnagar and Williams 1992, Clayton et al 1998, DFID 1995b, IDS 1998, Marsden and Oakley 1990, Marsden et al 1994, McGee and Norton 2000, Oakley 1988, Rudqvist and Woodford-Berger 1996, Uphoff 1989).

In particular, the cost-effectiveness of participatory approaches in development projects is difficult to determine, because economic and social parameters are only partly adequate in measuring costs and benefits (FAO 1990). Another difficulty is that participatory methods are very context sensitive and participation is affected by a wide range of factors (Martin and Sherington 1997).

Objectives of participation and hypotheses of its costs and benefits

OBJECTIVES

Approaches to monitoring and evaluating participation in projects and programmes differ according to what is being measured, which is turn depends on the objectives of participation. Although the objectives vary according to the type and context of the particular development project or programme, it is possible to identify some broad objectives of participation.

Bhatnagar and Williams (1992) and Oakley (1991) summarize some common objectives of participation as:

Other objectives are:

Campbell and Salagrama (2000) categorise the objectives of participation under the following headings:

Different areas of development efforts emphasize different aspects of these broad goals. For example, in a participatory and integrated watershed management project, the objectives regarding the process of participatory implementation were (Warren 1998):

In the area of participatory research, McAllister (1999) distinguishes two main goals or rationale for choosing a participatory research approach in projects. These may be functional or empowering or a combination of both. The functional goal of participatory research is to involve local people in order to improve the effectiveness and usefulness of the research for the community. For example, involving farmers in research to develop improved farming technologies should increase the appropriateness and enhance the adoption of the technologies. The empowering goal of participatory research is to strengthen individual and collective capacity and decision-making power in the wider society.

HYPOTHESES

There are a number of hypotheses about the costs and benefits of stakeholder participation in development projects and programmes and on the impact of stakeholder participation on the performance and outcomes of these projects and programmes. These are based on observations, experiences and case studies of participatory development activities over the years.

Benefits

One of the main purposes of participation in development is to improve effectiveness of development efforts. It is hypothesized that projects will more likely achieve their objectives if they have been identified, designed, implemented and evaluated with the participation of the people most affected by them (IDB).

Participation can be expected to improve the chances of aid being effective because, in drawing on a wide range of interested parties, the prospects for appropriate project design and commitment to achieving objectives is likely to be maximized (DFID 1995c). Participation should also improve the chances of a project being sustainable because people are more likely to be committed to carrying on the activity after aid stops, and more able to do so given that participation itself helps develop skills and confidence. (Bhatnagar and Williams 1992, Rudqvist 1992).

It is generally assumed that participation of rural people in the different stages of development projects and programmes will be cost-effective in the long run. The essence of participatory approaches in the project cycle is to promote self-reliance and it is assumed that this will result in decreasing recurrent costs and increasing cost-recovery by the project participants (FAO 1990).

Clayton et al (1994) summarize widespread hypotheses of the benefits of participation in rural development projects and programmes. It is expected that participation can:

Uphoff (1989) speaks about the benefits of participatory self-evaluation for the groups involved in the People's Participation Programme (PPP). These are:

In the area of participatory poverty assessments, it is hypothesized that participatory approaches can increase the breadth, quality and relevance of information at the disposal of policy makers (McGee and Norton 2000).

McAllister (1999) reports that the potential benefits of participatory monitoring and evaluation include:

The Inter-American Development Bank expects that participation will improve project design, help resolve or manage conflict, generate social learning and invention and strengthen local institutions. Participation is also seen as contributing to the goals of good governance, respect for human rights and democratisation (IDB). More specifically, the Bank sees the potential benefits of participation as:

According to the World Bank (Bhatnagar and Williams 1992), participatory approaches should allow governments to:

Fragmentary evidence from the FAO People's Participation Programme shows that participation in rural development has benefits for other individuals and society as a whole. Benefits to individuals include: increased food production, higher net family incomes, increased employment, higher rates of savings, acquisition of new skills. Benefits to society as a whole include: creation of zero-cost receiving systems inasmuch as delivery of services to organized small farmers is more cost efficient and rural delivery systems become self-propelling, building of rural community infrastructure at low-cost, strengthening of rural institutions (FAO 1990).

In the realm of policy, it is expected that "broad-based participation of civil society in the adoption and monitoring of the poverty reduction strategy tailored to country circumstances will enhance its sustained implementation" (McGee and Norton 2000, p. 8).

Costs

Among the costs of participation, time costs are frequently mentioned. Participatory methods of project design and implementation require more time than standard procedures (Clayton et al 1994, DFID 1995c, IDS 1998, World Bank 1996). This can be costly for the beneficiaries as well as for the donor agencies. Time poverty is something that often affects the rural poor, particularly women, who can ill afford to take time away from production to engage in meetings and other participatory processes (McGee and Norton 2000).

Another cost involved regards training. Participatory processes require training at all levels from villagers to senior management. Staff of donor agencies need to be trained in using participatory methods and skilled facilitators and other resource persons must be identified and involved (IDS 1998).

Participation in projects and programmes is likely to entail financial costs. This can include upfront costs to the donor and other stakeholders involved in identifying and engaging with each other (Clayton et al 1994, IDB, World Bank 1996).

Costs to World Bank borrowing countries include increased time for consultations and the cost of extension services, particularly the recurrent cost of fielding community promoters or change agents (World Bank 1996).

Participation entails opportunity costs for all the stakeholders (IDB). These may be particularly high for the beneficiaries, especially marginal groups and women, and may add to the work burden or decrease in leisure time. Potential costs could also include the risk that participation may not yield expected results and benefits, or even that it may unintentionally cause harm to the community or specific groups within the community (McAllister 1999).

It has also been observed that popular participation tends to be local cost-intensive, rather than foreign cost-intensive and to take a longer amount of time to design and implement (Rudqvist 1992). A hidden cost to the local beneficiaries is the shifting of the burden onto the poor by national governments of their responsibilities to promote development with equity (Clayton et al 1994).

Special attention has been given to the costs required for participatory monitoring and evaluation. In addition to financial costs for the donor, there are time costs for both the project beneficiaries and the project staff. Participatory M&E requires greater degrees of coordination, administrative efforts and long-term commitment on the part of stakeholders at all levels. Costs also include the training to acquire the skills for carrying out participatory M&E (Abbot and Guijt 1998, Estrella and Gaventa).

Costs of participation are usually weighed against estimated benefits. Uphoff (1992) suggests, however, that the costs of participation should be weighed against the costs of not encouraging and assisting participation. It is hypothesized that low levels of popular participation may lead to high costs later because of low impact or project failure. Such costs include the lack or misuse of facilities, poor maintenance, deterioration of infrastructure, and programme decline (Bhatnagar and Williams 1992, Uphoff 1992).

Approaches to evaluating stakeholder participation

In order to evaluate stakeholder participation in development projects and programmes, it is first necessary to identify the stakeholders. This section looks at who the stakeholders are and how to identify them. It then examines approaches and methods for assessing: the extent and quality of stakeholder participation; the costs and benefits of participation to each stakeholder group; and the impact of stakeholder participation on project and programme performance, outcomes and sustainability.

While there is considerable attention in the literature to identifying stakeholders and assessing the extent and quality of stakeholder participation, there is very little about assessing the costs and benefits of participation to the stakeholders or the impact of participation on performance, outcomes and sustainability. There is not always a clear separation among the approaches and methods for assessing these different aspects of participation. Some that are described under one heading have been or could be used for another aspect of assessing participation as well.

WHO ARE THE STAKEHOLDERS?

Initial efforts to promote participation in development focused on popular participation and on the rural poor and other disadvantaged people because, although they are usually the intended beneficiaries, the poor and disadvantaged are often excluded from having a voice in the development process. Increasingly, attention is also being given to other stakeholders and the emphasis is shifting from popular participation to stakeholder participation (World Bank 1996).

Stakeholders are those who are affected by the outcome, negatively or positively, or those who can affect the outcomes of a proposed intervention. In development projects and programmes, stakeholders are commonly divided into primary and secondary stakeholders:

In addition to primary and secondary stakeholders, there may be external stakeholders, i.e. people and groups not formally involved in a project, but who may impact or be impacted by the activity (DFID 1995a, Clayton et al 1996). Stakeholders are sometimes also categorised according to their relative importance or influence. Importance refers to the extent that the needs and interests of stakeholders are priorities of a project. Influence refers to the power stakeholders have over the project (Grimble and Wellard 1997). Among primary and secondary stakeholders, some will be key stakeholders, i.e. those who can significantly influence the project, or are most important for meeting the project objectives (DFID 1995c).

The following table gives examples of broad categories of stakeholders as identified by some multilateral agencies:

UNDP
(Clayton et al 1996)
World Bank (1996) Inter-American Development Bank (IDB) World Bank Poverty Reduction Strategy
(McGee and Norton 2000)
UNDP. The Bank, i.e. Bank management, staff and shareholders. The IDB. Donor agencies.
Other donors.   Other development agencies.  
National government. Borrowers, i.e. the governments and government officials representing borrower member countries. National government; municipal and state governments. Central government; local-level governments officials.
Civil society organizations, including NGOs. Indirectly affected groups, e.g. NGOs, private sector organizations and others with an interest in the outcomes. Local, national, and international NGOs; other civil society groups; private sector and professional groups that may mobilise or contribute resources to the project. Civil society organizations representing poor sectors (e.g. Church leaders, trade unions, farmers' unions, development NGOs); academic researchers; politicians and political parties; the communication media; the non-poor.
Local communities / beneficiaries. Directly affected groups, i.e. those who stand to benefit or lose from Bank-supported operations. These often include the poor and the marginalised. Representative community-based organizations (CBOs); traditional, informal social networks; informal or temporary coalitions or social networks arising due to project intervention. People in poor communities and their associations.

Identifying stakeholders

There are various methods for identifying the relevant stakeholders.

Guiding questions

The World Bank (1996) suggests some guiding questions for identifying stakeholders:

Contrast or maximum variation sample

A method for identifying stakeholder groups in the community is the use of a "contrast" or "maximum" variation sampling procedure. This can be used to define local groupings around issues. Each individual interviewed is asked to identify another who will have the most different perceptions on the issue than his / her own. The process of interviewing and identifying new respondents with contrasting views is repeated until several main issues or themes emerge. These themes each represent a stakeholder group. This approach enables the identification of groups with conflicting or different values without asking direct questions that may be socially unacceptable to answer (McAllister 1999).

Stakeholder Analysis

Stakeholder analysis is commonly used to identify all the groups and individuals who have a stake, or vested interest, in the success or failure of a project or activity (DFID 1995a, Clayton et al 1996). According to DFID (1995a), "stakeholder analysis is the identification of a project's key stakeholders, an assessment of their interests, and the ways in which these interests affect project riskiness and viability."

The first step in a stakeholder analysis is to identify and list all potential stakeholders. These can be listed under the headings of primary, secondary and external stakeholders. Primary stakeholders can be categorised according to social analysis and divided by gender, social or income classes, occupational or service user groups. There may be some overlap among the categories of primary stakeholders (e.g. women and low income groups, or minor forest users and ethnic minorities). Secondary stakeholders can be divided into funding, implementing, monitoring and advocacy organizations, or into governmental, NGO and private sector organizations. These categories may need to be further sub-divided. For example, key individuals may have personal interests at stake as will as formal institutional objectives; organizations may have sub-groups which should be considered as stakeholders (DFID 1995a).

A checklist (DFID 1995a) for identifying stakeholders includes the questions:

New stakeholder groups may be created as a result of the intervention of a project that brings new or additional resources to an area (DFID 1995c).

Assessing the extent and quality of stakeholder participation

The FAO People's Participation Programme (PPP) emphasized that evaluation of stakeholder participation is:

  1. concerned with processes which are qualitative and not results that are quantitative; and
  2. more concerned with description and interpretation than with measurement and prediction (Marsden and Oakley 1984).

The measurement of participation requires:

  1. valid criteria for understanding the nature of participation in a rural development project;
  2. a set of indicators to give form to these criteria;
  3. appropriate methods at project level for monitoring the indicators and maintaining a continuous record of the process of participation; and
  4. interpretation of the information recorded in terms of making a judgment concerning participation (Marsden and Oakley 1984).

Because traditional monitoring and evaluation has been concerned with quantifiable measurements, there is a new focus on the qualitative aspects of participation and on the process of participation. However, both qualitative and quantitative aspects of participation are important (Clayton et al 1994, Marsden and Oakley 1994, Oakley 1988, Oakley 1991, Rudqvist and Woodford-Berger 1996s).

This requires two forms of monitoring and evaluation (Oakley 1988):

Because participation is a dynamic process that must be evaluated over time, conventional ex post evaluations are inadequate. Ongoing monitoring is the only way qualitative descriptions can be obtained over time. It should be participatory, involving the rural people involved in the project.

Key characteristics to this qualitative approach to evaluating stakeholder participation are described as (Clayton et al 1994, Oakley 1991):

The steps involved are:

Indicators of participation

There are no generic indicators of participation. The indicators selected will vary according to the project and its objectives. Bhatnagar and Williams (1992) propose three very broad categories of indicators:

Other development authors and agencies have drawn up categories or questions that can be used in developing indicators of the extent and quality of participation. The following table gives two examples:

Quantifiable indicators

Oakley 1991 DFID 1995b
  • Economic indicators of participation. These measure
    1. the economic benefits of the project, and
    2. who is participating in the project's benefits, an analysis of the sectors of the rural population who have directly benefited and a quantitative assessment of the benefits on their lives and their future ability to sustain the level of activities.
  • Organizational indicators of participation. In projects in which organizations play an important role as a vehicle for participation, it is important to develop indicators to measure the extent of rural people's participation in organizations. These could include such things as: percentage of rural adults in the project areas who are members of the organization, frequency of attendance and changing size of membership over time. These indicators could be disaggregated as regards gender and socioeconomic groups.
  • Participation in project activities: These measure the numbers and kinds of groups of people involved in the project in different ways.
  • Development momentum: These include such things as numbers of project members aware of and in contact with development agencies' services, numbers receiving training, links with similar projects.

Who is participating?

How many people are participating and through what institutional arrangements?

Are local project institutions developing satisfactorily?

Project input take-up rates - are people actively engaged in the project?

What is the level of participation in key activities?

Are participants mobilising their own resources and contributing to the project materially?

Are installations kept in good running order by the participants?

Qualitative indicators

Oakley 1991 DFID 1995b
  • Organizational growth: e.g. internal structuring of project group, leadership, formalisation of group structure.
  • Group behaviour: e.g. changing nature of involvement of project group members, emerging sense of collective will and solidarity, involvement in discussions and decisions, ability to analyse and explain issues and problems.
  • Group self-reliance: e.g. increasing ability to propose and consider courses of action, knowledge and understanding of policies and programmes, changing relationship with project staff, formalisation of independent identity, independent action.

How are groups expected to achieve stability?

What capabilities are participating groups being encouraged to develop?

What are the expected qualities of participants' contributions?

What behavioural characteristics are groups and participants expected to display?

Are groups achieving increased self-reliance and control?

In addition to quantitative and qualitative indicators DFID (1995b) stresses the importance of the time dimension of participation. This refers to the timing of activities, such as the formation of groups, the local production of micro-plans, the farmers' on-site trial results, etc. The time dimension can be measured by the sequence of activities rather than by time limits; e.g. by looking at the sequence of the formation of farmers' groups, rather than setting a time limit within which so many groups should be formed.

When possible the intended beneficiaries should be involved in developing both quantitative and qualitative indicators of participation. While it may be difficult to involve primary stakeholders in the identification of these indicators at the design stage of a project, it may be possible to use participatory processes during the project implementation to help the primary stakeholders to generate their own indicators (DFID 1995b, 1995c).

Once the indicators are selected, they must be monitored and recorded around a series of pre-determined aspects of the project, e.g. project activities, changes in group behaviour, group action and articulation, relationship with the project. Monitoring is a critical dimension of qualitative evaluation. A range of monitoring methods and instruments can be used for collecting the data, as well as forms for presenting the interpretation of the data and information collected (Oakley 1988, DFID 1995b). Oakley (1988, 1991) presents lists of possible indicators, methods, instruments, and forms.

Interpretation is another critical dimension of the evaluation of qualitative indicators of participation. UNDP recommends that this be a participatory exercise and involve both project group members and staff. This can take place in open discussion or review meetings under the guidance of a facilitator and be structured around the indicators. Interpretation should take place regularly, every three months or so (Clayton et al 1994).

There is an increasing recognition of the need for standards for quality in participation. Two issues arise in setting standards: what benchmarks are needed to judge quality and how should these benchmarks be established. There are no universal benchmarks, as these are context specific. Nevertheless, it is important to set some non-negotiable standards, so as not to risk being pushed down to the lowest common denominator. The process of developing quality standards should be participatory and involve the primary stakeholders. In practice people's participation in setting of standards has been low, however, partly because of their perceptions that the costs of this participation outweigh the benefits (McGee and Norton 2000).

Participant observation and group discussions

Assessing the quality of participation can benefit from participant observation and the recording of selected qualitative information on e.g. who speaks at meetings, does one person or group dominate the discussion, how are decisions made and conflicts managed. In addition to field observation, local people can be asked to provide feedback through various participatory methods (McAllister 1999). Three clusters of questions are suggested:

Participant observer evaluation is conducted by a neutral qualified observer, but it allows for considerable participation by different stakeholders (Rudqvist and Woodford-Berger 1996).

Gender analysis

In measuring the extent and quality of participation, it is important to understand how gender differences affect the participation of women in development activities. Gender analysis should, therefore, be an integral part monitoring and evaluation activities. Research on the gender dimensions of poverty has brought to light a major constraint that affects the extent and quality of participation in development projects: time poverty. Rural people, and women in particular, often lack the time to take part in meetings and decision-making, especially if this takes them away from production. Consequently, payment for participants' time has come to be recognised as a factor in raising the quality of participation (McGee and Norton 2000).

Participation matrix

DFID (1995c) proposes using a participation matrix for negotiating participation in projects. Used throughout the project cycle, this matrix could also help identify the extent and quality of stakeholder participation. The matrix is based on the following range of possible stakeholder participation:

  Inform Consult Partnership Control
Identification        
Planning        
Implementation        
M&E        

The different stakeholders, from senior levels in donor and recipient institutions, to the primary stakeholders at the local level are entered in the boxes according to how they participate at the different stages of the project cycle. This matrix needs to be revised during the course of the project as participation changes.

Survey

The Inter-American Foundation (IAF) views NGOs as effective vehicles for delivering direct development assistance. NGOs, however, vary greatly in their management style and cannot be presumed to be participatory in their relations with their target groups. IAF concern for the participation of stakeholders has led it to support several studies in recent years to evaluate NGO performance in IAF-funded projects. The extent and quality of participation of the stakeholders and accountability of NGO leadership and staff were important criteria taken into consideration.

A 1992 study examining 30 organizations funded by IAF used the following questions as a guide in evaluating their performance with respect to participation:

Participation in this study was viewed as both an input and an output of NGO activity. Participation as a process was evaluated by rating local stakeholder involvement along four dimensions:

  1. initial decisions,
  2. implementation,
  3. accountability, and
  4. resource mobilisation.

Participation as a result was seen as an increase in the local stakeholders' capacity to influence decision-making in a systematic, normalised way (IDB).

ASSESSING THE IMPACT OF STAKEHOLDER PARTICIPATION ON PROJECT AND PROGRAMME PERFORMANCE, OUTCOMES AND SUSTAINABILITY

Reviews of ex post evaluations

One way that donor agencies have attempted to evaluate the impact of stakeholder participation is through studies of ex post evaluations. There are two examples in the literature of such studies conducted by the World Bank.

Cernea (1991) reports on an analysis of 25 World Bank-financed projects that were re-evaluated several years after the financing was terminated, in order to assess the long-term sustainability of these projects. Thirteen of the projects were found to be non-sustainable. Although the primary reason was insufficient financing, the lack of farmers' organizations and participation of the primary stakeholders in project formulation and implementation was considered to be a contributing factor.

Kottak (1991) reports on an analysis of 68 ex post evaluation reports of the World Bank, consisting of 57 Performance Audit Reports and 11 Impact Evaluation Reports to uncover the main lessons regarding the socio-cultural variables of Bank-assisted rural development projects. Most of the experiences showed the tendency to address technical and financial factors and to neglect social issues. The most significant finding was that attention to social issues resulted in economic benefits. Projects that were socio-culturally compatible and based on an adequate understanding and analysis of the social conditions had average rates of economic return that were more than twice as high as those for socially incompatible and poorly analysed projects.

The analysis covered projects with a regional distribution mainly in Africa, Asia, Latin America and the Caribbean. Sectoral coverage included irrigation, agricultural and multi-sectoral development, settlement, livestock and fisheries. The projects were coded as socio-culturally compatible or not based on both explicit statements in the primary documents and on the analyst's identification of a lack of fit between project goals or implementation strategy and local culture. Factors deemed to affect social sensitivity and cultural fit included: formal and informal social organizations of the project area population; issues of stratification, ranking and ethnicity; gender roles and division of labour; and systems of values and motivations. The socio-cultural dimensions of the projects were quantified by variables and coding in each case. Among the specific cultural and socio-economic variables considered were:

The report on this analysis stresses the importance of a socially sensitive perspective and data collection in evaluation and the role of social expertise in the ex post evaluation of impact. It asserts that this social expertise does not need to come from a foreign anthropologist or sociologist. It can be provided by national experts and by project participants themselves.

Case studies

One of the most common ways of examining the impact of participation on project and programme performance is through case studies. Sometimes these case studies are based on project appraisal and / or evaluation documents.

One approach is to analyse why projects were not as successful as expected and to deduce where participation of beneficiaries, particularly in the design and implementation stages, might have helped avoid some of the mistakes made (Uphoff 1991).

Another approach was taken in analysing the economic, social and environmental impacts of participatory watershed development (Hinchcliffe et al 1995). A two-year collaborative study provided detailed case studies of 22 participatory watershed development projects in Asia, Africa, Latin America and Australia. The case study teams were all already engaged in participatory approaches to watershed development, but few had experience in impact assessments. The indicators were designed to be both quantitative and qualitative and to provide information on the benefits and costs as perceived by the local people.

Six set of indicators were drawn up, based on a literature review. The indicators were grouped as follows:

Participatory methods were used in preparing the case studies, wherever possible. The case studies reported on the economic, environmental and social impacts of the projects. The social benefits included greater self-confidence and sense of cohesion in communities, reduced conflicts over resources, reduced out-migration, attention to the needs of landless groups and new rapport between local people and external professionals.

Following the case studies, the team members met in a workshop to discuss lessons learned from the case studies and to suggest ways forward regarding: technologies, process and methods, impacts and indicators, inter-institutional arrangements and policies. The lessons relevant to evaluation of the impact of participation were:

Surveys

An example of how surveys can be used to assess the impact of participation on project and programme outcomes is provided by Abbot and Guijt (1998). This was a survey of the positive and negative impacts of participatory monitoring processes involving over 200 community environmental monitoring (CEM) experiences in Australia. It was conducted through national surveys and workshops. The questions included:

The responses showed a positive assessment of CEM and highlighted some requirements for more fully realising its potential.

Statistical analysis

A statistical analysis of 121 rural water supply projects in Asia, Africa and Latin America, supported by 18 international agencies, found that "beneficiary participation" was the single most important factor in determining overall quality of the project implementation. Reported in Rudqvist and Woodford-Berger (1996), this analysis considered beneficiary participation to be significant in contributing to project effectiveness, to the proportion of water systems in good condition, overall economic benefits, percentage of target population reached, and environmental benefits. Other results of beneficiary participation were strengthened community organizations and the acquisition of new skills.

Etic and emic approaches

Borrowing from anthropology, Uphoff (1992) categorises methods of monitoring and evaluating participation under two approaches: the etic and the emic. The etic approach attempts to look at reality in objectified terms and includes methods such as standard monitoring and evaluation techniques, participant observation and beneficiary assessment, rapid rural appraisal, participatory rural appraisal and focus groups. The emic approach is an insider look at reality as understood by people in their own culture or society. It includes methods such as local planning, monitoring and evaluation, participatory self-evaluation, visitation and peer training, and monitoring and evaluation across levels. According to Uphoff, monitoring and evaluation of participation should be undertaken from both perspectives.

Uphoff gives an example of how standard monitoring and evaluation techniques have been used in practice in a project to establish water-user associations in a run-down irrigation scheme in Gal Oya, Sri Lanka. This example also serves as a caution about the limitations of "objective" techniques. The project officials decided to use just a few, but appropriate, indicators. Two indicators were used to measure group strength and extent of participation:

  1. frequency of meetings and
  2. voluntary group efforts.

Two measures of impact were also monitored:

  1. number of complaints about water distribution and
  2. number of untouched and unbroken channel gates.

In terms of impact, the number of complaints dropped nearly to zero and the number of broken gates dropped from about 80 percent to nearly zero. In terms of participation, it was found that the number of field groups meeting at least monthly declined from about 80 to 50 percent over two years and that there was a similar drop in the proportion of voluntary group efforts to clean irrigation channels. The evaluation concluded that organizational capacity was declining.

This conclusion was contested by the farmers and the organizers. According to the farmers, the reason for the decline in frequency of meetings was the good functioning of the groups. The groups had functioned so well at the beginning of the project that most of the problems had been solved and there was less need to meet as often. Moreover, there was now greater participation at higher levels. As regards voluntary group labour, once the channels had been cleaned by group effort, it was easier and more convenient for farmers to maintain them in satisfactory condition partly on an individual basis. In fact, the farmer's organizations had asked the district administrative officer to put pressure on individuals to maintain their own field channels. This itself is an indication of group solidarity and strength.

This experience shows that indicators of participation should not remain static, but evolve over time as the conditions change. Evaluations must reconsider the meaning of their indicators in terms of progress and performance in the face of changing participation.

Participatory monitoring and evaluation

The FAO People's Participation Programme (1990) developed participatory evaluation of self-reliance and progress towards self-sustainability. In this approach group monitoring and ongoing evaluation is undertaken through group meetings in which all members try to reach consensus in ranking the group's progress, using a number of self-selected self-reliance variables. Indicators could include such things as regularity of group meetings, shared leadership and member participation in group decision-making, continuous growth in group savings, high rates of loan repayment, group problem solving, and effective links with development services. Group promoters develop a group self-reliance monitoring system, based on a review of the group's record books, group promoter meetings to review progress, and periodic sample surveys conducted with randomly selected groups. Record, or log, books keep a running record of group activities, decisions and membership. Experiences with these methods showed that small-scale farmers' organizations or informal groups were more successful when members organized around solving common problems identified by themselves, than when groups organized around problems identified by outsiders (Huizer 1983).

Another example of participatory monitoring and evalaution is cited in Abbot and Guijt (1998): in Brazil four stakeholder groups collaborated to assess the change brought about by their participation in collective activities in sustainable agriculture. The four stakeholder groups were: women and men farmers, representatives of the rural workers' union, local NGO staff, and academics. These stakeholder groups met, usually in mixed groups, to develop a monitoring methodology:

In using participatory M&E to evaluate participation, it is necessary to decide which stakeholders should be involved and at which stages of the evaluation. One way of doing this is to look at each of the indicators and decide which stakeholder group should be involved by asking the following questions: What is the relevance of participating for each group? In which stage of the evaluation? Who is going to use the final information? How difficult is the calculation?

In Sri Lanka, a bottom up approach was introduced for planning, monitoring and evaluation for improved irrigated agriculture (Uphoff 1992). This approach simplified the goals and limited them to three: efficiency of water use, productivity of resource inputs, and profitability of farming operations. These could be measured with some simple numbers, i.e.:

Productivity = yields per acre by crop (measured at end of season).

Profitability = net income in rupees per acre, by crop and average.

Before a season, farmers' groups agreed on targets and plans for reaching these. The plans were compiled and passed on to a project committee consisting of farmer representatives and officials. At the end of the season, the farmers' groups calculated their performance in terms of efficiency, productivity and profitability and considered where and why they had either fallen short or surpassed the targets. The groups monitored a limited number of activities that were expected to help achieve the targets: these included both participatory and non-participatory activities.

Ideas from an approach for evaluating participatory research

Some of the issues involved in evaluating participatory research may be relevant for assessing the impact of stakeholder participation on projects and programmes. McAllister (1999) outlines the characteristics of an approach for evaluating participatory research as follows:

McAllister (1999) suggests drawing on a number of evaluation approaches, including:

Measuring social transformation and empowerment

If empowerment is one of the goals of participation, then some ways should be developed to evaluate whether or not this has occurred. For this, empowerment must be defined and indicators of empowerment developed. McAllister (1999) suggests that indicators of empowerment encompass personal as well as socio-economic and political change and can be established at the levels of individuals, groups or communities. Criteria for determining whether participation encourages social change and contributes to local empowerment can be categorised in the following clusters:

Some ideas from a review of donor agency documents

A review of documents from eight bilateral donor agencies (DANIDA, DGIS, FINNIDA, GTZ, NORAD, ODA (DFID), SIDA and USAID) led to the following general conclusions on how to evaluate participation (Rudqvist and Woodford-Berger 1996). Evaluation of participation should be done:

ASSESSING THE COSTS AND BENEFITS OF PARTICIPATION TO EACH STAKEHOLDER GROUP

Experiences in assessing the costs and benefits of participation to the different stakeholders have been limited. The few assessments documented in the literature pertain mainly to the costs to the donor agency. The World Bank, in particular, has carried out some assessments of the costs of participation to the Bank. Some of the literature suggests that participatory monitoring and evaluation could be used to assess the costs and benefits of participation to the primary stakeholders.

World Bank survey

Within the context of a larger World Bank study on participatory development, attention was given to assessing the costs and benefits of participation to the Bank (Bhatnagar and Williams 1992). The study, carried out by the Bank's Learning Group on Participation, covered 20 projects selected out of a pool of some 80 projects that were nominated as participatory. The Learning Group drew up 21 questions dealing with the Bank's capacity to support participation in Bank-financed projects and general issues relating to participation in borrowing countries. The survey questions relevant to assessing the costs of participation to the Bank were:

The responses from the project staff indicated that participatory projects often require more time in the preparatory stages than non-participatory projects in the same region and sector. More time was required for the consultative process, for identifying the groups and organizations that should be involved in the participatory process, for reaching agreement among various stakeholders on the meaning of participation, and for building trust among the stakeholders. There were also indications that time could be saved through using already existing organizations instead of attempting to form new ones. In some instances, more time invested in the preparation of participatory projects reduced the time needed in the implementation stage.

Responses to the question of cost indicated that the costs for preparing and implementing participatory projects may be no more than for non-participatory ones. However, costs of supervising some of the 20 participating projects in the survey were higher than the supervision costs in non-participatory projects.

In reporting on the results of this survey, Bhatnagar (1992) points out that the Bank should take into consideration not only costs of supporting participation in projects, but also the costs of not providing for participation, which might include lack or misuse of facilities, poor maintenance, deterioration of infrastructure and programme decline.

Survey of Swedish International Development Authority (SIDA)

Using the World Bank priority questions on popular participation, key SIDA staff with experience in various participatory projects were interviewed (Rudqvist 1992). The responses to the questions relevant to assessing the cost of participation to SIDA were:

Participatory projects tend to require more SIDA staff and consulting time, particularly for planning and preparation. In the field, more resources are needed for local-level consultations, organization, mobilisation and training. However, most participatory projects are low-cost projects that are labour-intensive rather than capital intensive. A considerable part of the cost is borne by the recipient country in terms of labour and material. The Survey results suggested that it would be useful to compare the costs and benefits of participatory projects with those of non-participatory approaches.

Cost and benefit mapping

A World Bank Workshop on participatory development came up with a proposal for cost and benefit mapping as an instrument for evaluating the costs and benefits of participation to different stakeholders (Bhatnagar and Williams 1992). The maps would show:

The Workshop also identified some of the cost and benefit parameters for the maps. Costs include financial and economic costs, training, consultation (within and between participating institutions), preparation work, and physical execution. Benefits include capacity created, changes in perception, design, commitment, implementation performance, resource mobilisation, sustainability, savings in time, and savings in lives.

Interviews and comparison with control group

Another study to determine the costs of participation to World Bank projects examined:

The study involved interviewing the staff of 21 participatory projects, computing standard statistics for a larger group of participatory projects, and comparing these to a Bank-wide control group using the Management Information System. Costs were interpreted in function of the World Bank's overall institutional objectives, i.e. as the incremental amount of resources invested in the preparation, appraisal, negotiation and supervision of participatory investment projects when compared to non-participatory projects. This is appropriate if the Banks sees itself as an institution that minimizes its costs regardless of project quality. However, if the Bank aims to finance sustainable, high quality projects and if participatory methods ensure that quality, then costs would have to be compared to alternative methods to achieve the required project quality.

The 21 projects selected for analysis all included beneficiary or stakeholder participation at the decision-making level in the implementation phase of the project. The project staff were asked what they considered to be the net cost of participation, i.e. not to quantify the time and money needed to conduct the participatory process, but to compare overall costs of preparation and supervision of the participatory project to a hypothetical non-participatory project. The main questions asked were:

For the comparison with the Bank-wide control group, the number of participatory projects was supplemented with others for a total of 42 projects. The control group projects were selected from active or completed World Bank projects. Three different questions were asked:

The results of the surveys showed that:

Participation mainly affected the salary budget: there were higher salary expenses for Bank staff and consultants as working weeks increased in both the preparation and implementation phases. The facilitation costs of the preparatory process were borne mainly by the borrowers or the beneficiaries.

Participatory projects cost about 10 to 15 percent more to finance staff and consultants during participation. Two-thirds of this was for the inclusion of participation specialists.

Participatory projects required more intensive supervision during the early implementation stage. The supervision of participatory projects averaged 27 staff weeks compared to 17 staff weeks for the control group.

Participation was partly financed by trust funds. Participatory projects used extra-budgetary funds to finance about 15 percent of staff and consultant input, while the control group used these funds for about 5 percent of this input.

Participation did not require a longer time span for presentation to the Board or to become effective. Elapsed time was slightly longer for preparation and appraisal only.

The study suggested that preparation and supervision costs would decline over time as staff learn to apply participatory techniques. Further monitoring of the supervision effort would be needed to see whether costs decrease over the implementation phase. Finally, it asserted that the costs of participation should be evaluated in combination with project quality. Whatever the definition of project quality, if participation is a means to achieve this goal, it may turn out to be the least cost solution in the long run.

Participatory monitoring and evaluation

Participatory project evaluation by stakeholders is one way to assess whether the benefits of a project have been distributed equitably. There is an extensive literature on participatory monitoring and evaluation, focusing mainly on methodologies for conducting this (Abbot and Guijt 1998, Estrella and Gaventa, IDB, Huizer 1983, Oakley 1988, 1994, Rudqvist and Woodford-Berger 1996, Uphoff 1982). Some of the literature stresses the importance of using participatory monitoring and evaluation for the monitoring and evaluation of participation (Oakley 1988, 1991). However, there are few concrete examples of how this has been or could be done specifically for measuring the costs and benefits to the different stakeholder groups.

Gender issues

In measuring the costs and benefits to primary stakeholders, it is necessary to be gender-specific. Because of the differential access to resources and constraints to participation on men and women, costs and benefits may be different. It is not possible to assume that women will automatically share in benefits to primary stakeholder groups (World Bank 1996).

Conclusions

Although participation is now widely accepted as an important element in development projects and programmes, the monitoring and evaluation of participation is still in its infancy. There is a vast amount of literature dealing with participation in development, most of it focusing on conceptual issues of participation or on participatory techniques, including participatory monitoring and evaluation. Literature dealing with the monitoring and evaluation of participation is still very limited and documentation of experiences in monitoring and evaluating of participation is more limited still. This reflects the limited experience of bilateral and multilateral development agencies in monitoring and evaluating participation.

There are great expectations of the benefits of participation in development projects and programmes. It is hypothesized that participation will improve the quality and increase the efficiency, effectiveness, sustainability and coverage of projects and programmes. Promotion of participation is also expected to benefit the primary stakeholders, building their capacity and leading to self-reliance and empowerment. These hypotheses are not based solely on theoretical considerations, but are the fruit of accumulated experiences both in the promotion of participation and in the failure of non-participatory development efforts.

A number of difficulties are encountered in monitoring and evaluating participation:

  1. participation is understood in vastly different ways in development projects and programmes;
  2. participation impacts not only the project or programme but also the different stakeholders in different ways;
  3. participation is both a means to help achieve project efficiency, effectiveness, coverage and sustainability and an end to promote stakeholder capacity and empowerment and, therefore, must be monitored and evaluated from these two different perspectives; and
  4. participation is a qualitative process.

Monitoring and evaluation of participation requires measuring quality, both the quality of participation and the impact of participation on the quality of the project. Conventional monitoring and evaluation methods that are oriented to quantitative measurements are inadequate to the task.

These difficulties are not insurmountable, however. Considerable work has already been done to overcome the obstacles to monitoring ands evaluating participation. First of all, there has been a clear identification of the need to monitor and evaluation three different aspects of participation:

In most development projects and programmes, participation is considered to be both a means and an end. Conceived of as an end, participation needs to be evaluated as to extent and quality. Conceived of as a means, participation needs to be assessed as to its impact on the performance, outcomes and sustainability of the project or programmes. It is also important to know the extent and the quality of the participation that led to its impact. Finally, there is a need to assess the costs and benefits of participation to each of the stakeholder groups in order to determine whether the promotion of participation in projects and programmes is cost effective.

There is a great deal of theoretical work, much of it based on practical experience, on the definitions and concepts of participation. Participation can be understood along a continuum ranging from contribution of inputs to predetermined projects and programmes, to information sharing, consultation, decision-making, partnership and empowerment. Different stakeholders participate in different ways at different stages of the project cycle. There is a sufficient body of theoretical work and the practical experience to enable development agencies to be clear about what they mean by participation when undertaking participatory projects and programmes or introducing participation into a project or programme.

A clear understanding of participation at the outset of a project or programme facilitates monitoring and evaluation of participation. On the other hand, participation is still sometimes introduced into a project or programme without a clear understanding of what this entails. This lack of clarity makes it difficult to monitor and evaluate participation.

A considerable amount of work has also been done to identify who the stakeholders are in projects and programmes. The initial focus on beneficiaries has expanded to include a wider range of stakeholders, i.e. all those who are affected by the outcome, negatively or positively, or those who can affect the outcomes of a proposed intervention. Stakeholders can be categorised as: primary stakeholders, those people and groups who are ultimately affected by the project; secondary stakeholders, intermediaries in the process of delivering aid to primary stakeholders; and external stakeholders, those not formally involved in a project, but who may impact or be impacted by it. In development projects and programmes, stakeholders usually include donor agencies, government, civil society organizations and the local community and beneficiaries. Stakeholder analysis is one of the major methods used for identifying the relevant stakeholders of a particular project or programme. Techniques for stakeholder analysis are well developed and relatively simple to use.

While there is considerable attention in the literature to identifying stakeholders and assessing the extent and quality of stakeholder participation, there is very little about assessing the costs and benefits of participation to the stakeholders or the impact of participation on performance, outcomes and sustainability. However, there is not always a clear separation among the approaches and methods for assessing these different aspects of participation.

The most well-developed and in-depth approach to assessing the extent and quality of stakeholder participation is that developed by and for FAO's People's Participation Programme. The focus is on the participation of the primary stakeholders and is based on a process of ongoing monitoring and evaluation involving beneficiaries, group promoters and project staff. Sets of quantitative and qualitative indicators have been developed to assess the extent and quality of participation. This approach requires considerable time investment by both the primary stakeholders and project staff. However, a number of techniques have been developed to reduce the time costs.

The FAO PPP approach was developed in the 1980s and later efforts by other development agencies appear to build on or adapt this approach to their needs. DFID, for example, has developed a list of questions that examines both the quantitative and qualitative aspects of primary stakeholder participation. In addition, DFID stresses the importance of measuring the time dimension of participation as well as the quantity and quality. UNDP also recommends a participatory approach similar to the PPP approach. UNDP suggests that the ongoing monitoring and evaluation can take place in open discussion or review meetings under the guidance of a facilitator and be structured around the indicators. All of these approaches stress the need for interpretation of the data and suggest a range of methods, instruments and forms for collecting data.

Participant observation and group discussions is another proposed approach. Participant observer evaluation is conducted by a neutral qualified observer, but it allows considerable participation by the different stakeholders. In addition to field observation, this approach can include asking stakeholders to provide feedback through various participatory methods. This approach also requires sets of quantitative and qualitative indicators and questions.

Most of these approaches also stress the importance of gender disaggregation and understanding gender differences in assessing the extent and quality of participation. It would be worthwhile exploring how socio-economic and gender analysis could be used in assessing the extent and quality of participation.

The main focus of these approaches is on the primary stakeholders. One technique that could be adapted to show the extent and quality of participation by the whole range of stakeholders relative to each other is DFID's participation matrix. This matrix is intended for use in the early stages of a project to negotiate participation among the various stakeholders. Used throughout the project cycle, it could help map the changing extent and quality of participation by all stakeholders.

Ideally, monitoring and evaluating of the extent and quality of participation should be an ongoing process carried out throughout the life of the project or programme. When this has not been done, development agencies use questionnaires, survey and case studies to assess the extent and quality of participation at different points in the life of a project or after it has terminated.

While there are many examples of monitoring and evaluation of the outcomes, performance and sustainability of participatory projects and programmes, few of these have explicitly examined the impact of participation on the project relative to other factors.

One of the approaches to assessing the impact of participation is the review of ex post evaluations to assess the long-term sustainability of participatory versus non-participatory projects. A similar approach was used by the World Bank in reviewing ex post evaluations to compare the impact of projects that gave attention to social issues to projects that did not. This approach included coding the projects on the bases of socio-cultural variables as discerned in explicit statements in the primary project documents and in the analyst's identification of a fit between project goals and implementation and local culture. Such an approach could be adapted to look specifically at stakeholder participation. The limitations of this approach are that it depends on the quality of the ex post evaluations and the information that is included in these documents, as well as on the judgment of the analyst. Costs in terms of staff time and/or consultants' fees would depend on the extent of the review.

One of the most common approaches to assessing the impact of participation is through case studies. These case studies can be based on appraisal and/or evaluation documents or be undertaken during the implementation stage of the project. Case studies can use both quantitative and qualitative indicators and participatory methods. Workshops following the case studies have been used to discuss the lessons learnt and to apply these to ongoing projects and to the planning of new ones. The advantages of undertaking case studies during the life of the project is the possibilities of gathering insights from a range of stakeholders and for making adjustments to ongoing activities.

Questionnaires and surveys are another way to gather information on the impact of participation. These can range from simple to complex and cover a range of stakeholders.

Standard monitoring and evaluation techniques have been used to measure the project outcomes and performance and the extent and quality of participation and to compare the two. It is possible to use just a few, but appropriate, indicators for this purpose. A caution has been raised in using this approach, in that participation evolves over the life of a project. It is, therefore, necessary to reconsider the meaning of indicators in terms of progress and performance in the face of changing participation.

Participatory monitoring and evaluation involving primary stakeholders can also be used to assess the impact of participation. Participatory M&E techniques are highly developed and are commonly used to enable the primary stakeholders to make their own assessments of project outcomes and performance. Usually these exercises do not explicitly examine how participation has affected the outcomes and performance compared to other factors. This, however, could be added to participatory monitoring and evaluation approaches.

Experiences in assessing the costs and benefits of participation to the different stakeholders have been limited. The few assessments documented in the literature pertain mainly to the costs to the donor agency. The World Bank, in particular, has carried out some assessments of the costs of participation to the Bank. The approaches used were surveys of project staff, interviews and comparison with a control group of projects. A World Bank workshop also suggested cost and benefit mapping that would show benefits and costs as seen by different stakeholder groups; benefits and costs tracked over time; and the causal linkages between benefits and costs.

Some of the literature suggests that participatory monitoring and evaluation could be used to assess the costs and benefits of participation to the primary stakeholders.

The review of the literature shows that while experiences in monitoring and evaluating stakeholder participation are still limited, a body of experience and knowledge is being accumulated. The challenge for development agencies is to build on this experience and knowledge in order to use, adapt and develop appropriate approaches to monitoring and evaluating stakeholder participation, apply these systematically, and utilize the lessons learnt.

References

Abbot, Joanne and Irene Guijt (1998) Changing Views on Change: Participatory Approaches to Monitoring the Environment, SARL Discussion Paper No. 2, London: International Institute for Environment and Development (IIED).

Bhatnagar, Bhuvan ( 1992) "Participatory Development and the World Bank: Opportunities and Concerns", in Bhatnagar, Bhuvan and Aubrey C. Williams (eds.), Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper 183, Washington D.C.: The World Bank.

Bhatnagar, Bhuvan and Aubrey C. Williams (eds.) (1992) Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper 183, Washington D.C.: The World Bank.

Campbell, Jock and Venkatesh Salagrama (2000) New Approaches to Participation in Fisheries Research, Rome: FAO and SIFAR.

Cernea, Michael M. (ed.) (1991) Putting People First: Sociological Variables in Rural Development, New York: Oxford University Press.

Clayton, Andrew, Peter Oakley and Brian Pratt (1998) Empowering People: A Guide to Participation, New York: UNDP.

Department for International Development (DFID) (1995a) Guidance Note on How to Do Stakeholder Analysis of Aid Projects and Programmes, London: DFID.

Department for International Development (DFID) (1995b) Guidance Note on Indicators for Measuring and Assessing Primary Stakeholder Participation, London: DFID.

Department for International Development (DFID) (1995c) Technical Note on Enhancing Stakeholder Participation in Aid Activities, London: DFID.

Edgerton, J., K. McClean, C. Robb, P. Shah and S. Tikare, (April 2000) "Participatory Processes in the Poverty Reduction Strategy" in Poverty Reduction Strategy Sourcebook, Draft for Discussion, Washington D.C.: The World Bank.

Estrella, Marisol and John Gaventa (no date) Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review, IDS Working Paper 70, Brighton: IDS.

Food and Agriculture Organization (FAO) (1990) Participation in Practice: Lessons from the FAO People's Participation Programme, Rome: FAO.

Grimble, Robin and Kate Wellard (1997) "Stakeholder Methodologies in Natural Resource Management: A Review of Principles, Contexts, Experiences and Opportunities", in Agricultural Systems, Vol. 55, No. 2.

Hentschel, Jesko (1994) Does Participation Cost the Bank More? Emerging Evidence, Human Resources Development and Operations Policy Working Papers 31, Washington D.C.: World Bank.

Hinchcliffe, Fiona, Irene Guijt, Jules N. Pretty and Parmesh Shah (1995) New Horizons: The Economic, Social and Environmental Impacts of Participatory Watershed Development, Gatekeeper Series No. 50, London: International Institute for Environment and Development (IIED): London.

Huizer, Gerrit (1983) People's Participation Projects: Guiding Principles, Rome: FAO.

Institute of Development Studies (IDS) (1998) Participatory Monitoring and Evaluation: Learning from Change, IDS Policy Briefing Paper Issue 12, Brighton: IDS.

Inter-American Development Bank (IDB) (no date) Resource Book on Participation, Washington D.C.: IDB.

Kottak, Conrad Phillip (1991) "When People Don't Come First: Some Sociological Lesson From Completed Projects", in Cernea, Michael M. (ed.), 1991, Putting People First: Sociological Variables in Rural Development, New York: Oxford University Press.

Marsden, David and Peter Oakley (eds.) (1990) Evaluating Social Development Projects, Oxford: Oxfam.

Marsden, David, Peter Oakley and Brian Pratt (1994) Measuring the Process: Guidelines for Evaluating Social Development, Oxford: INTRAC.

Martin, Adrienne and John Sherington (1997) "Participatory Research Methods - Implementation, Effectiveness and Institutional Context", in Agricultural Systems, Vol. 55, No. 2.

McAllister, Karen (1999) Understanding Participation: Monitoring and Evaluating Process, Outputs and Outcomes, Ottawa: IDRC.

McGee, Rosemary and Andy Norton (2000) Participation in Poverty Reduction Strategies: A Synthesis of Experience with Participatory Approaches to Policy Design, Implementation and Monitoring, IDS Working Paper 109, Brighton: IDS.

Oakley, Peter and David Marsden (1984) Approaches to Participation in Development, Geneva: International Labour Organisation (ILO).

Oakley, Peter (1988) The Monitoring and Evaluation of Participation in Rural Development, Rome: FAO.

Rudqvist, Anders (1992) "The Swedish International Development Authority: Experience with Popular Participation" in Bhatnagar, Bhuvan and Aubrey C. Williams (eds.), 1992, Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper 183, Washington D.C.: The World Bank.

Rudqvist, Anders and Prudence Woodford-Berger (1996) Evaluation and Participation -Some Lessons, SIDA Studies in Evaluation 96/1, Stockholm: SIDA.

Skutsch, Margaret M. (1998) "First Analyse Your Conflict", in Proceedings of a Satellite Meeting to the XI World Forestry Congress, Forest, Trees and People Programme, Conflict Management Series, pp. 261-305, Rome: FAO.

Uphoff, Norman (1992) "Monitoring and Evaluating Popular Participation in World Bank-Assisted Projects" in Bhatnagar, Bhuvan and Aubrey C. Williams (eds.), Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper 183, Washington D.C.: The World Bank.

Uphoff, Norman (1991) "Fitting Projects to People", in Cernea, Michael M. (ed.), 1991, Putting People First: Sociological Variables in Rural Development, New York: Oxford University Press.

Uphoff, Norman (1989) Participatory Self-Evaluation of P.P.P. Group and Inter-Group Association Performance: a Field Methodology, Rome: FAO.

Warren, Patrizio (1998) Developing Participatory and Integrated Watershed Management, Community Forestry Case Study Series 13, Rome: FAO.

World Bank (1998) Assessing Aid: What Works. What Doesn't, and Why, New York: Oxford University Press.

World Bank (1996) The World Bank Participation Sourcebook, Washington D.C.: The World Bank.



Back to Top FAO Homepage