Programme Committee

Table of Contents

Summary of Main Findings on Areas for Improvement

Introduction

Synthesis of Field Project Evaluations 1994-99 (Excluding TCP)

COMPARISON OF PRPJECTS: 1985-91 VS. 1994-99
EFFECTS AND IMPACT OF PROJECTS
POTENTIAL FOR IMPROVEMENT IN PROJECT PERFORMANCE

Quantitative Analysis from Mission Questionnaires

QUALITATIVE ANALYSIS OF MISSION FINDINGS AND RECOMMENDATIONS
FUTURE REPORTING TO THE GOVERNING BODIES ON SYNTHESIS OF PROJECT EVALUATIONS

Thematic Evaluations of TCP Projects:
Apiculture and Sericulture (1998) and Legislation (1999)

EVALUATION METHODOLOGY
THE EVALUATIONS
GENERAL LESSONS FOR FAO-TCP

Summary of Main Findings on Areas for Improvement

The analysis indicated that results were good in terms of achieving the projects' outputs and effects for the benefit of the target populations. Nevertheless, particular areas were identified for attention in project design and implementation. These include:

• More realism as to adequate project duration, especially in institution building and people's participation projects. Projects also need to be designed in such a way as to assure sustainable results with or without a second phase;
  
  
• Greater realism as regards national capacities, which tended to be over-estimated;
  
  
• Improvements in project design, especially for more concise and realistic objectives and clearer identification of beneficiaries. However, for FAO-TCP projects, formulation missions should not generally be undertaken, as this was unlikely to be cost-effective;
  
  
• In institutional strengthening of government capacity, more attention to increasing capacity of existing units and creating new ways of working across institutional boundaries to tackle emerging problems, instead of creating new units or expanding staff;
  
  
• In policy and planning, increased attention to the identification of issues and to the facilitation of dialogue between the community and the political level;
  
  
• More attention to economic and marketing aspects in projects for development of production and those for people's participation;
  
  
• Greater attention to linkages and collaborative working arrangements, especially in ensuring the sustainability of efforts for people based development;
  
  
• For TCP projects, greater attention to project closure and follow-up, including through a formal meeting to consider project results and follow-up action;
  
  
• For TCP careful consideration before funding projects in specialised technical areas which may not be in areas of the Organization's comparative advantage and where technical support is not adequate.
  
  
• Despite the considerable improvement evidenced in project design, this is still the major cause of poor results and therefore further efforts should be pursued in the context of decentralized arrangements for the operational activities, including in particular:
• preparation of updated guidelines in project formulation and design;
• further training of the FAO staff, especially those in the decentralized offices, on the subject; and
• strengthening the existing mechanisms for reviewing and appraising project proposals, both at the decentralized offices and at Headquarters.

Introduction

1. The note brings together an analysis of the findings of field project evaluations of UNDP and trust fund projects by independent missions and thematic evaluations of FAO Technical Cooperation Programme (TCP) projects.

2. This is the fourth report to the Governing Bodies covering the results of individual field project evaluations since the 1992-93 Programme Evaluation Report (PER). In that document (C 93/4, Chapter 4), a synthesis was made of all project evaluations in the period 1985-91. Subsequent reports on the results of project evaluations were contained in the Programme Implementation Report (PIR) 1994-95 (covering evaluations from 1992-94) and the PIR 1996-97 (covering evaluations through 1997). This report for 1994-99 includes the time periods reported in the previous PIRs and is updated to include the biennium 1998-99. The year 1994 is appropriate as the starting point for the study because in that year a number of improvements were introduced in the FAO evaluation process, including specific requirements to address the issues of cost-effectiveness and greater focus on effects and impact and contributing factors to project sustainability.

3. The note includes quantitative information, as presented in the PIRs, but also includes more in-depth analysis of factors associated with effects and impact and qualitative assessments based on reports covering particular subject areas. The focus is on finding areas for improvement, particularly in order to achieve project effects and impact. The analysis is based on the findings of independent tripartite evaluation missions for 279 projects evaluated between 1994 and 1999¹. In terms of areas of work, the projects covered were in reasonable harmony with the distribution of FAO's overall field programme (see Table 1). Theme-oriented TCP² evaluations began in 1997, with a study of food quality control projects, the results of which were reported in the Programme Evaluation Report 1998-99 (Chapter Four). This report covers apiculture and sericulture (1998) and assistance in legislation (1999). Groups of TCP projects are evaluated by the Evaluation Service around particular themes, in order to draw lessons for future design of similar projects, determine achievements at project level, improve cost-effectiveness and provide a basis for accountability reporting.

4. The Technical Cooperation Department, in consultation with the Evaluation Service, makes the choice of topics for evaluation. Apiculture and sericulture were selected for evaluation in order to address a "problem area" and given the large volume of requests. While important for small farmers in many parts of the world, these topics represent a very small part of FAO's overall programme. Nonetheless, a large number of requests for TCP projects in these areas had been received and approved over the three biennia preceding the study. A few of these projects had notable implementation difficulties that were known to the Technical Cooperation Department before the study was undertaken. Assistance in legislation is one of the major areas for TCP requests and this was the main reason to select this topic. The evaluation examined not only projects that were exclusively in the legal sphere, but also reviewed projects in other subject areas that had a legal component.

Synthesis of Field Project Evaluations 1994-99
(Excluding TCP)

Table 1: Coverage of All Evaluations excluding TCP (1994-1999) by Technical Programme Area in Relation to FAO's Total Field Programme

5. Table 2 below presents statistical information on evaluations undertaken by region and source of funding. The Table also presents global information on the types of evaluation undertaken (at mid-term, terminal, or ex-post).

Table 2: Coverage of all Evaluations (1994-99) by Region, Source of Funding and Type of Evaluation (in percentage)

COMPARISON OF PROJECTS: 1985-91 VS. 1994-99

6. As noted above, project evaluation report formats were amended in 1994 to be more comprehensive in their coverage and to better reflect concerns expressed by the Governing Bodies. Table 3 examines the results of projects over the last three biennia against those that were reported to the Governing Bodies in the PER 1992-93.

Table 3: Overall Assessment of Project Evaluations (excluding TCP)

7. The results point to an improvement over time. Considerable effort has been put into improving project design, for example, by upgrading staff skills through training programmes and improving programming and evaluation guidelines. Equally important have been the stringent standards established by donors for accepting proposals for funding, including requirements for project design. Similarly, project effects are increasingly rated "good".

EFFECTS AND IMPACT OF PROJECTS³

8. Table 4 attests to very reasonable results from projects in terms of their benefits to countries, as determined by evaluation missions. It is against these basically positive findings that subsequent analyses of areas for improvement need to be viewed.

Table 4: Overall Summary of Project Performance (1994-99)

9. Only 18 evaluations (6%) were ex-post. Thus, most evaluations were looking forward towards probable eventual impact but effects (use of outputs by beneficiaries) were frequently already evident, especially for projects evaluated at or near the end of project implementation (terminal evaluations). There was some divergence between regions (see Annex Table 1). In particular, missions had lower expectation for use of outputs and sustainable impact in projects evaluated in the Near East and Europe. Although the reasons for this were not uniform, missions found that project designs consistently showed extreme over-optimism as to what could be achieved within project budgets and time-frames and also over-estimated the capacities of national staff⁴.

10. Projects were examined for impact against different types of objectives, i.e. a) Policy, planning and legislative improvements b) Strengthening national institutional capacity c) Uptake of technical improvements d) Expansion of pilot activities and e) Follow-up investment. Follow-up investment was the weakest and uptake of technical improvements and institution building the strongest. As might be expected, the risk of no impact was greatest in projects involving uptake of policy planning and legislative improvements, replication of pilot activities and investment follow-up. In such projects, achievement of impact usually depends on positive decisions at higher echelons, beyond where the project may have been working (if there is no follow-up, there can be little impact).

POTENTIAL FOR IMPROVEMENT IN PROJECT PERFORMANCE

Quantitative Analysis from Mission Questionnaires

11. The analysis below focuses on the identification of potential areas for improvement, based on evaluation missions' assessments on the quality of project selection, design and management. It also examines the specific factors that contributed most to limiting project effects and impact, and cost-effectiveness.

Table 5: Areas for Improvement (1994-99) - Mission Assessments of Project Selection, Design and Implementation Management

Table 6: Aspects of Project Design and Implementation Identified by Missions as most contributing to sub-optimal performance and where there could be improvements (1994-99)

Project Design

12. Project Design: Quantitative analysis showed that project design was the weakest aspect of the projects evaluated: missions reported in 33% of cases that design was the aspect of the project where there was the greatest need for improvement (Table 6). As can be seen from Table 5, missions found that project designs suffered from over-optimism as to what could be accomplished in the project duration. Fifty-one percent of projects were considered too short and analysis showed lack of realism on duration was co-related with reduced effects and impact. In projects performing below the optimum in terms of effects and impact, analysis showed formulators were particularly over-optimistic as to the use that could be made of project outputs. Projects also tended to have unrealistic expectations of the capabilities and resources of national institutions (Table 6). Further, there was inadequate attention to risks and prerequisites for project success (Table 5).

13. In separate questions on factors having the greatest negative impact on project cost-effectiveness, scheduling was found to have been the most important factor in 21% of cases (Table 6). There tended to be over-expectation of what inputs governments could reasonably be expected to provide to the project (Table 5). Other faults were less pervasive, but significant proportions of projects could have been considerably improved with respect to clarity of immediate objectives and targets. There was also found to be a need for more focus in projects in 39% of cases (Table 6). In further analysis, failure to adequately specify beneficiaries was found to be linked to sub-optimal performance in terms of effects and impact.

14. Missions found that 20-40% of projects would have been more cost-effective had there been more reliance on national training, more use of the private sector, NGOs, national experts and short-term staff and a greater reliance on government capacities. However, it was noticeable that there could also be too much of these features and it appears that where for instance, there was a heavy use of NGOs, short-term staff or government capacity, this was considered counter-productive (Table 6).

15. Project Implementation and Management: Projects which performed sub-optimally in terms of effects and impact suffered particularly from inadequate management, with internal management being the weakest point but FAO supervision also being an area for improvement. Both Government and FAO procedures were found by missions to have constrained performance and insufficient delegation of authority was identified as a problem in 30% of cases (Table 6). In what can be both a design and implementation fault, the involvement of beneficiaries could have been improved in 18% of cases (Table 6).

QUALITATIVE ANALYSIS OF MISSION FINDINGS AND RECOMMENDATIONS

16. Some of the issues identified above were amplified by a qualitative review of evaluation findings and recommendations, in particular project areas where there was a substantial volume of similar findings.

17. Capacity-Building: Institution strengthening projects evaluated in recent years indicate that the focus has been shifting increasingly from establishing or enlarging government departments to strengthening capacity of existing institutions for new functions, such as community forestry or for environmental planning. Missions were not generally optimistic for the sustainability of results of earlier institutional expansion projects due to shortages of government resources. More recent projects have been designed to develop people's self-reliance and ownership through participatory and group approaches. For both types of institution building, missions generally concluded that the project duration was too short and a further phase was essential to consolidate results. This points to:

• a need for donors to accept realistic durations in their commitments to institution-building;
• projects to be designed in such a way that even if they are terminated at the end of the project life with no extension, sustainable results are achieved.

18. Policy support: The relatively weak performance of projects in the area of policy support in realising their effects and impact has been noted above. Projects attempting to strengthen policy and planning directly were most influential not through direct provision of advice, but rather when they identified the major issues and supported national dialogue between the community and the political level. It was also found that most policy outputs were not from projects specialised in policy but from projects carrying out institutional development and development support. Thus a project for area development might have an influence on land tenure policy, one on community forestry might influence the approach to extension.

19. People's Participation: Projects where people's participation was important were on occasion criticised for giving excessive attention to establishing new participatory groups, while failing to successfully initiate other improvements through the groups. There were also criticisms that groups were not necessarily the best development tool in some situations and small-scale private initiative could have been encouraged as an alternative. As with production projects, unless people saw a definite benefit to themselves from the groups (usually economic), progress was limited. People's participation projects were sometimes found to be both too short and to be spread over an excessively wide area. This tends to indicate that participatory projects should initially work intensively in a small area with a few groups, thus leaving a lasting capacity in the groups themselves and in the support personnel who have worked with the groups.

20. In institution-building, people's participation and policy projects the qualitative analysis found linkages needed to be strengthened. There was a particular danger of projects in people's participation working as separate entities rather than forging sustainable local partnerships with NGOs and government agencies concerned. Similarly, missions found training to be among the projects' most valuable outputs for capacity-building. There appeared to be an underlying assumption that individuals with increased skills would contribute to development, even if the specific endeavour for which they were trained was not sustainable.

21. Development of Production and Land Management: For projects designed for the development of production and improved land management, there were frequent findings that insufficient attention had been given to economic and marketing aspects. There were a few examples of pilot demonstrations failing to achieve impact because of lack of a legal basis, particularly lack of clear tenure to the resource. There were also a large number of calls by missions for improved monitoring and evaluation to determine which approaches and packages were in reality proving acceptable to people and were producing the expected improvements. Similarly a few missions highlighted attention to these aspects as having been important factors in achieving sustainable effects and impact.

22. Gender: While there were not a large body of findings on reasons for the effectiveness of gender related activities, several missions noted a failure of projects to specifically target women, even when they were the main actors, and others pointed to success when packages and extension training were specifically designed for women. In some situations the use of female staff to reach women had been advantageous, even in societies where there were no particular taboos on men communicating with women.

23. Regional Projects: Some qualitative conclusions emerged with respect to regional projects, in particular:

• many regional projects had established networks for technical exchange. There was little confidence that these networks would be viable once the projects came to an end. The only network which was found definitely sustainable was for an association of commercial businesses in the seed sector. This does not invalidate networking during projects as a way of cost-effectively achieving outreach but it does indicate that undue energy should not go into trying to establish enduring technical exchange networks;
  
  
• in Asia there was a lack of integration in the activities of projects supporting various aspects of policy and planning development for forestry and these and other regional projects could have been more closely coordinated with FAO's regular programme activities and priorities.

24. Overall Conclusions and Recommendations: The present report shows considerable improvement overall in project design, implementation and production of outputs from the last examination over a period of time (1985-91). Nonetheless, it will be noticed that most of the findings and issues arising are very similar to those made in the PER 1992-93. It is recommended that given the persistent weaknesses in project design, further efforts should be pursued in the context of decentralized arrangements for the operational activities, including in particular: (a) preparation of updated guidelines in project formulation and design with particular attention to the areas needing improvements; (b) further training of the FAO staff, especially those in the decentralized offices, on the subject; and (c) strengthening the existing mechanisms for reviewing and appraising project proposals, both at the decentralized offices at at Headquarters.

FUTURE REPORTING TO THE GOVERNING BODIES ON SYNTHESIS OF PROJECT EVALUATIONS

25. This report also presented a limited examination of the qualitative aspects of particular types of projects and drew some conclusions in these various areas. It is suggested that for the immediate future, reporting on results of evaluations goes further in this direction and that syntheses of evaluation reports be based on particular programme areas and cross-cutting themes in line with the priorities of the Strategic Framework or possibly on the work in particular regions. Programmatic evaluations will also systematically assess the Regular Programme and related field activities. This approach would be consistent with that followed for evaluation of TCP. It is believed that such analysis, if based upon a sufficiently large sample of projects, will provide more focussed analysis and recommendations for consideration by Management and the Governing Bodies. A broader analysis may be repeated every 2 or 3 biennia when a further substantial body of evaluations have been completed.

Thematic Evaluations of TCP Projects:
Apiculture and Sericulture (1998) and Legislation (1999)

EVALUATION METHODOLOGY

26. Both evaluations were implemented following similar methodologies. In keeping with FAO policy, evaluations were carried out with the assistance of external consultants. Two consultants (one each for apiculture and sericulture) were used in the 1998 exercise and four consultants, each specialised in the region being visited, took part in the 1999 legislation review. For the apiculture/sericulture review, all projects (22) approved in the previous three biennia were evaluated while for legislation, a sample of 31 (34%) projects in 17 countries was reviewed.

THE EVALUATIONS

Apiculture and Sericulture

27. Apiculture and sericulture are potentially significant sources of income for small farmers in developing countries. Apiculture in particular can be important for the landless and is highly significant to agricultural production and ecosystem conservation in general due to its role in pollination. Sericulture is an important economic activity particularly for women, as they are most involved in the labour-intensive work of placing mature silkworms in frames and spinning yarn.

28. The majority of the projects dealt with significant development problems, usually relating to disease control (varroa in bees, pebrine disease in silkworms). However, some were not well justified on technical grounds or would not have met TCP criteria for approval had conditions on the ground been better known. Most of the projects were found to be stand-alone efforts and not part of any larger Government or donor-funded programme.

29. Four of the projects were considered to have had serious implementation difficulties. Performance of international consultants, with a few exceptions, was good. Nonetheless, it was noted that economies could have been achieved in some projects (particularly in sericulture) which used several experts with similar qualifications. Also, national consultants used in some countries tended to mirror the expertise of international consultants (although some were lacking in technical qualifications). Technical backstopping from FAO for the sericulture projects was found to be generally better than for apiculture. A serious problem was the late or non-production of terminal statements or letters of completion.

30. Evaluation missions rated the individual projects overall, taking into account all aspects and against the following criteria, using a 3-point scale (Good, Satisfactory, Unsatisfactory).

• Relevance: How pertinent and significant was the project's objective/purpose to the identified problems and needs?
• Design: How clear were the objectives and their links with inputs, activities and outputs? Was there a proper mix of inputs to achieve the desired results at the lowest cost? How effective was the institutional framework for the project?
• Implementation: Were inputs delivered and outputs produced on time and effectively? Was project management (including technical backstopping) effective?
• Effects/Use of outputs: Once produced, were outputs used as intended?
• Impact/Sustainability: Is the project likely to lead to production of the desired developmental results? Are results likely to be maintained/sustained at the conclusion of TCP assistance?

31. The results were as follows⁶:

Table 7: Assessments of Apiculture and Sericulture TCP Projects

32. The number of unsatisfactory projects was felt to be too high. On the whole, the best apiculture projects were those that dealt with disease outbreak (varroa) in countries where apiculture is economically significant. However, several apiculture projects were deemed unsuccessful because there was little scope for spreading their results among beekeepers in the country or their technical justification was poor. Because of much lower costs of entry, apiculture is likely to be of more interest than sericulture to most small farmers. It was found, however, that projects should be made more relevant to the majority of beekeepers in countries, rather than aiming at the most modern production methods. A consequence of this approach is that training did not have the desired impact at producer level.

33. Sericulture could be a relevant topic for TCP assistance, provided there is a plan for development of the sector and the role of the TCP project is clearly established. The evaluation found that chances for sustainable impact of sericulture projects are much less in countries with a limited tradition of sericulture and where there were no marketing opportunities.

34. It was found that there was insufficient linkage of lessons from projects to normative work at FAO and inadequate transfer of lessons from one project to another.

Legislation

35. The projects reviewed were found to be highly relevant to development problems in the recipient countries and the assistance received from FAO was greatly appreciated. Project design tended to be rather general in terms of description of problems to be tackled and the approach taken to implementation, but there are rather limited prospects for improving the amount of information made available. Any deficiencies in project design were generally remedied by the initial mission to the country, when issues were identified in greater depth and project implementation strategies decided.

36. Quality of the international consultants and Legal Office (LEG) staff (who served as legal consultants in 13 of the projects reviewed) was generally very high. The fact that FAO has a great deal of international expertise in various aspects of natural resources legislation was found to be its primary comparative advantage and the main reason for TCP project requests. Technical backstopping of legal components of projects has been generally excellent, with active interchange between LEG officers, consultants and national counterpart staff. In some cases, there has been continued post-project follow-up by individual LEG officers as legislation has worked its way through national processes. This was greatly appreciated by member governments.

37. Using the criteria mentioned for the apiculture/sericulture evaluation, legal projects (or legal components of projects) were scored as follows:

Table 8: Assessments of Legislation TCP Projects

38. While most projects were relevant and well-implemented, eventual project impact was found less satisfactory. In most countries where follow-up and impact were not satisfactory, the reason could be found in particular national circumstances such as economic upheaval or the failure to advance drafted legislation, which had been found acceptable by the individual Department concerned at the time of project implementation, through the legislative system. In order to ensure a complete package of assistance, the evaluation recommended that implementing regulations should normally be prepared in projects along with laws.

39. The two most important factors for likely project success were a high degree of stakeholder involvement in project implementation and a favourable policy framework for achieving the desired legislative reforms. Most projects made considerable effort to ensure broad consultation with potential stakeholders. Some notably successful projects also included a policy component and in these projects, there was close association between the legal and policy activities. Factors found that could negatively impact on project success included replacement of the project's main promoter or "champion", projects which are asked to mediate internal disputes and projects where legislative components are add-on measures outside the main project thrust. It also must be appreciated that proposed legislative reforms that seek to alter long-standing administrative arrangements often encounter resistance.

GENERAL LESSONS FOR FAO-TCP

40. The Evaluation Service has now conducted three thematic evaluations of TCP projects and some general recommendations and lessons have emerged which are highlighted here:

• Project follow-up requires improvement. In most cases (but not all), once project implementation ends, contact is lost with the Government counterpart body and Governments do not report to FAO on follow-up action taken, despite a requirement to do so in the TCP procedures.
• Although it will not completely solve the problem, the evaluation made proposals to improve the situation and these have been accepted by the TCP management. It has been agreed to hold a meeting at or near the end of the implementation period of all TCP projects to decide on follow-up action. The meeting will be called by the FAOR (where there is one) and be attended by the government counterpart agency and other interested parties, including bilateral or multilateral financing agencies that could be interested in assisting with project follow-up. It was agreed that the meeting be held around a paper prepared by or under the supervision of the responsible technical unit at FAO, which should ideally also attend. It is believed that a more formal consideration of project results and recommended follow-up will lead to improvements and facilitate post-project reporting on actions taken. This closure meeting will form an integral part of project reporting and closure.
• Evaluation missions have made numerous observations on project formulation. The criticisms in the last two evaluations were rather different. In legislation, project implementation strategies tended to be described in terms that were considered too general, while in apiculture and sericulture there was an evident disconnect in some projects between the described and actual situation in the field. This called into question the appropriateness of the concerned projects. One way often proposed to overcome these difficulties is investment of additional resources for project formulation. The evaluations examined whether formulation missions would be cost-effective. In the case of legislation projects, the conclusion was quite clear that project design deficiencies are usually corrected with the first consultancy mission and the present policy of TCP not to fund formulation missions was generally endorsed.
• Design of the apiculture/sericulture projects presented a more fundamental problem. While the Legal Office has staff resources to cover all technical areas and languages of FAO, this was not the case for apiculture and sericulture. For most of the period under review, there was one Headquarters officer in the Agro-Industries and Post-Harvest Management Service (AGSI) who looked after apiculture and sericulture projects. This raises priority and capacity questions as to the degree that FAO can execute activities in specialized areas where it does not have a sufficiently high level of technical support.

Extent to Which Project addressed a genuine development problem

Annex Table 3 was compiled based on a review of mission reports for qualitative lessons in predicting likely achievement of effects and impact. It includes those factors that had a particularly high percentage of scores in the range of 1 to 3 for effects and impact (column b), i.e. the range of nil to moderate effects and impact. Column (c) compares these scores with the percentage of scores (also in the range of 1 to 3, i.e. poor to satisfactory but not good) that the factors had in the total population of projects (column a). The bigger the difference in column (c), the more closely the factor in question is a predictor of sub-optimal effects and impact.

____________________________

1  A total of 301 projects were evaluated and reports met standards for comprehensiveness and quality of assessment in 279 cases (93% of the total).

2  The TCP is part of the Organization's Regular Budget, funded by assessed contributions from member states. TCP projects are of short duration (not over 24 months) with a maximum budget of $400,000 and are aimed at solving urgent problems faced by developing member nations of FAO. The projects are intended to be catalytic and to result in significant follow-up action by the Government.

3  Analysis hereafter refers to the period 1994-1999.

4  While the results for the Near East and Europe are considerably below those for other regions, they are based on a much lower sample of projects.

5  For this particular question, missions are offered only 3 choices: too short (51%), too long (3%), about right (46%).

6  Two of the projects evaluated were continuations of previous projects; thus the total number of results in each column will be 20, rather than 22.

7  too early in project implementation to assess