31 . This chapter provides the fourth report to the Governing Bodies covering the results of field project evaluations since the launch of the Programme Evaluation Report (PER) in 1993. An analysis is made of the findings of independent evaluations carried out during 1994-99 for individual UNDP and Trust Fund projects, as well as of thematic evaluations of TCP (Technical Cooperation Programme) projects, made by the Evaluation Service.
32 . The year 1994 is an appropriate starting point because in that year, a number of improvements were introduced in the FAO evaluation process, including specific requirements to address the cost-effectiveness and greater focus on effects and impact and factors contributing to the sustainability of results. The analysis is based on the findings of independent tripartite evaluation missions for 279 projects.
33 . Table 2.1 demonstrates an improvement over time. Considerable effort has been put into improving project design, for example, by upgrading staff skills through training and improving programme evaluation guidelines. Equally important have been the stringent standards established by donors foraccepting proposals for funding, including requirements for project design. Similarly, project effects are increasingly rated as “good”.
Table 2.1: Overall Assessment of Project Evaluations (1994-99) excluding TCP
| Aspect | Percentage rated 1985-91 | Percentage rated 1994-99 | ||||
| Poor | Satisfactory | Good | Poor | Satisfactory | Good | |
| Project Design | 26 | 56 | 18 | 21 | 40 | 39 |
| Implementation | 29 | 49 | 22 | 9 | 39 | 52 |
| Project Outputs: | ||||||
| quality | 9 | 62 | 29 | 7 | 43 | 50 |
| quantity | 13 | 57 | 30 | 8 | 44 | 48 |
| Project Effects | 16 | 60 | 24 | 11 | 39 | 50 |
34 . Table 2.2 attests to very reasonable results from projects in terms of benefits. There was some divergence between regions. In particular, missions had lower expectations for use of outputs and sustainable impact in projects evaluated in the Near East and Europe. Although the reasons for this were not uniform, missions found that the designs of these projects consistently showed extreme over-optimism as to what could be achieved within budgets and timeframes and also over-estimated the capacities of national staff. 6
Table 2.2: Overall Summary of Project Performance (1994-99) excluding TCP
| Projects were found to have: | ||||
| Addressed a genuine development problem | A problem of major importance | 85% | A significant problem | 98% |
| Effects in terms of use expected to be made of outputs | At least 80% of outputs expected to be used as foreseen | 50% | At least 60% expected to be used as foreseen | 89% |
| Expectations of sustainable impact | Considerable impact | 45% | Some or more sustainable impact | 86% |
| Cost-effectiveness for sustainable effects | The most cost-effective approach | 73% | ||
35 . Projects were examined for impact against different types of objectives, i.e. a) policy, planning and legislative improvements; b) strengthening national institutional capacity; c) uptake of technical improvements; d) expansion of pilot activities; and e) follow-up investment. Follow-up investment was the weakest, whereas uptake of technical improvements and institution-building was the strongest. As might be expected, the risk of no impact was greatest in projects involving uptake of policy planning and legislative improvements, replication of pilot activities and investment follow-up. In such projects, achievement of impact usually depends on positive decisions at higher echelons, beyond where the project may have been working (if there is no follow-up, there can be little impact).
36 . Project design : Missions reported in 33% of cases that design was the aspect of the project where there was the greatest need for improvement. Fifty-one percent of projects were considered of too short duration and analysis showed lack of realism on duration was co-related with reduced effects and impact. In projects performing below the optimum in terms of effects and impact, analysis showed formulators were particularly over-optimistic as to the use that could be made of project outputs. Projects also tended to have unrealistic expectations of the capabilities and resources of national institutions. Further, there was inadequate attention to risks and prerequisites for project success.
37 . Among factors having the greatest negative impact on project cost-effectiveness, scheduling was found to have been the most important factor in 21% of cases. There tended to be over-expectation of what inputs governments could reasonably be expected to provide to the project. Other shortcomings were less pervasive, but significant proportions of projects could have been considerably improved with respect to clarity of immediate objectives and targets. There was also a need for better focus in 39% of cases. Failure to adequately specify beneficiaries was found to be linked to sub-optimal performance in terms of effects and impact.
38 . Missions found that 20-40% of projects would have been more cost-effective, had there been: more reliance on national training; more use of the private sector, NGOs, national experts and short-term staff; and a greater reliance on government capacities. However, there were opposite cases where for instance, heavy use of NGOs, short-term staff or government capacity, was considered counter-productive.
39 . Project implementation and management : Projects which performed sub-optimally in terms of effects and impact suffered particularly from inadequate management, with internal management being the weakest point, but FAO supervision also being an area for improvement. Both government and FAO procedures were found to have constrained performance and insufficient delegation of authority was identified as a problem in 30% of cases. In what can be both a design and implementation problem, the involvement of beneficiaries could have been improved in 18% of cases.
40 . Capacity-building : Evaluation of institution-strengthening projects indicates that the focus has been shifting increasingly from establishing or enlarging government departments to strengthening capacity of existing institutions for new functions, such as community forestry or for environmental planning. Missions were not generally optimistic about the sustainability of results of earlier institutional expansion projects due to shortages in government resources. More recent projects have been designed to develop people's self-reliance and ownership through participatory and group approaches. For both types of institution-building, missions generally concluded that the project duration was too short and a further phase was essential to consolidate results. This points to: a need for donors to accept realistic durations in their commitments to institution-building; and for projects to be designed in such a way that even if they are terminated at the end of the project life with no extension, sustainable results are achieved.
41 . Policy support : Projects attempting to strengthen policy and planning directly were most influential when they identified the major issues and supported national dialogue between the community and the political level. Most policy outputs were not from projects specialised in policy but from projects carrying out institutional development and development support. Thus, a project for area development might have an influence on land tenure policy, and one on community forestry might influence the approach to extension.
42 . People's participation : Projects where people's participation was important were on occasion criticised for giving excessive attention to establishing new participatory groups, while failing to successfully initiate other improvements through the groups. There were also criticisms that groups were not necessarily the best development tool in some situations and small-scale private initiative could have been encouraged as an alternative. As with production projects, unless people saw a definite benefit to themselves from the groups (usually economic), progress was limited. People's participation projects were sometimes found to be too short and to be spread over too wide an area. This tends to indicate that participatory projects should initially work intensively in a small area with a few groups, thus leaving a lasting capacity in the groups themselves and in the support personnel who have worked with the groups.
43 . Institution-building, people's participation and policy projects: The qualitative analysis found linkages needed to be strengthened. There was a particular danger of projects in people's participation working as separate entities rather than forging sustainable local partnerships with NGOs and government agencies. Similarly, missions found training to be among the project's most valuable outputs for capacity-building. There appeared to be an underlying assumption that individuals with increased skills would contribute to development, even if the specific endeavour for which they were trained was not sustainable.
44 . Development of production and land management : For projects designed for the development of production and improved land management, there were frequent findings that insufficient attention had been given to economic and marketing aspects.
45 . Gender : Several missions noted a failure of projects to specifically target women, even when they were the main actors, and others pointed to success when packages and extension training were specifically designed for women. In some situations, the use of female staff to reach women had been advantageous, even in societies where there were no particular taboos on men communicating with women.
46 . Regional projects : Some qualitative conclusions emerged with respect to regional projects, in particular:
47 . The present report, covering the 1994-99 period, shows considerable improvement overall in project design, implementation and production of outputs from the last examination (1985-91). It is recommended that given the persistent weaknesses in project design, further efforts should be pursued in the context of decentralized arrangements for operational activities, including in particular: (a) preparation of updated guidelines in project formulation and design with particular attention to the areas needing improvements; (b) further training of FAO staff, especially those in the decentralized offices; and (c) strengthening the existing mechanisms for reviewing and appraising project proposals, both at the decentralized offices and at Headquarters.
48 . It is suggested that for the future, synthesis reporting on results of evaluations focus on particular programme areas and cross-cutting themes in line with the priorities of the FAO Strategic Framework 2000-2015, or possibly on the work in particular regions. Programmatic evaluations will also systematically assess the Regular Programme and related field activities.
49 . Apiculture and sericulture (22 projects) : The majority of projects dealt with significant development problems, usually relating to disease control (varroa in bees and pebrine disease in silkworms). However, some were not well justified on technical grounds or would not have met TCP criteria for approval had conditions on the ground been better known. Most of the projects were found to be stand-alone efforts and not part of any larger government or donor-funded programme.
50 . Performance of international consultants, with a few exceptions, was good. Nonetheless, it was noted that economies could have been achieved in some projects (particularly in seri-culture) which used several experts with similar qualifications. Also, national consultants used in some countries tended to mirror the expertise of international consultants (although some were lacking in technical qualifications). Technical backstopping from FAO for the sericulture projects was found to be generally better than for apiculture. A serious problem was the late or non-production of terminal statements or letters of completion.
51 . The number of unsatisfactory projects was felt to be too high. The best apiculture projects were those that dealt with disease outbreak (varroa) in countries where apiculture is economically significant. However, several apiculture projects were deemed unsuccessful because there was little scope for spreading their results among beekeepers in the country or their technical justification was poor. Because of much lower costs of entry, apiculture is likely to be of more interest than sericulture to most small farmers. It was found, however, that projects should be made more relevant to the majority of beekeepers in countries, rather than aiming at the most modern production methods. A consequence of this approach is that training did not have the desired impact at the producer level. Sericulture could be a relevant topic for TCP assistance, provided there is a plan for development of the sector and the role of the TCP project is clearly established. The evaluation found that the chances of a sustainable impact of sericulture projects were much reduced in countries with a limited tradition of sericulture and where there were no marketing opportunities.
52 . Legislation (31 projects ) : The projects reviewed were found to be highly relevant to development problems in the recipient countries and the assistance received from FAO was greatly appreciated. Project design tended to be rather general in terms of description of problems to be tackled and the approach taken to implementation, but there were limited prospects for improving the amount of information made available. Any deficiencies in project design were generally remedied by the initial mission to the country, when issues were identified in greater depth and project implementation strategies decided.
53 . The quality of the international consultants and Legal Office (LEG) staff (who provided advice in 13 of the projects reviewed) was generally very high. The fact that FAO has a great deal of international expertise in various aspects of natural resources legislation was found to be its primary comparative strength and the main reason for TCP project requests. Technical backstopping of legal components of projects has been generally excellent, with active inter-change between LEG officers, consultants and national counterpart staff. In some cases, there has been continued post-project follow-up by individual LEG officers as legislation has worked its way through national processes. This was greatly appreciated by the concerned governments.
54 . While most projects were relevant and well implemented, eventual project impact was found to be less satisfactory. In most countries where follow-up and impact were not satisfactory, the reason could be found in particular national circumstances such as economic upheaval or the failure to advance drafted legislation, which had been found acceptable by the individual department concerned at the time of project implementation. In order to ensure a complete package of assistance, the evaluation recommended that implementation regulations should normally be prepared in projects along with laws.
55 . The two most important factors for likely project success were a high degree of stakeholder involvement in project implementation and a favourable policy framework for achieving the desired legislative reforms.
56 . Project follow-up requires improvement. In most cases (but not all), once project implementation ends, contact is lost with the government counterpart body and governments do not report to FAO on follow-up action taken, despite a requirement to do so in the TCP procedures. FAO management has now agreed to hold a meeting at or near the end of the implementation period of all TCP projects to decide on follow-up action. The meeting will be called by the FAOR (where there is one) and be attended by the government counterpart agency and other interested parties, including bilateral or multilateral financing agencies that could be interested in assisting with project follow-up. This meeting will form an integral part of project reporting and closure.
57 . Evaluation missions have made numerous observations on project formulation. The criticisms in the last two evaluations were rather different. In legislation, project implementation strategies tended to be described in terms that were considered too general, while in apiculture and sericulture there was an evident disconnect in some projects between the described and actual situation in the field. This called into question the appropriateness of the concerned projects. The evaluations examined whether investment of additional resources in formulation missions would be cost-effective. In the case of legislation projects, the conclusion was quite clear that project design deficiencies are usually corrected with the first consultancy mission and the present policy of TCP not to fund formulation missions was generally endorsed.
58 . Design of the apiculture/sericulture projects presented a more fundamental problem. While the Legal Office has staff resources to cover all technical areas and languages of FAO, this was not the case for apiculture and sericulture. For most of the period under review, there was one Headquarters officer in the Agro-Industries and Post-Harvest Management Service (AGSI) who looked after apiculture and sericulture projects. This raises priority and capacity questions as to the degree that FAO can execute activities in specialized areas where it does not have a sufficiently high level of technical support.
(Report of the 83rd Session, May 2000 8 )
i. The Committee considered that this report provided a useful synthesis of assessments by independent project evaluation missions and those by the Evaluation Service on selected TCP projects.
ii. The Committee noted with concern that, despite the overall trend for improvement in the key aspects of field project performance, project design remained the weakest aspect. While noting some improvement in the percentage rated “good”, it expressed concern that weakness in project design had been persistently highlighted in similar evaluation syntheses over a number of years, and queried what corrective actions were being taken. The Committee recognized that the problem was complex because projects dealt with complex development issues over a wide range of sectors under differing conditions and because any corrective measures involved several units within the Organization. It nevertheless stressed the urgent need for concerted action in order to ensure the quality of project formulation, particularly in the context of changing procedures and arrangements for the Field Programme.
iii. In this connection, the Committee endorsed the lines of action regarding updating the project formulation guidelines, further training of FAO staff and strengthening the project review and appraisal mechanisms. In particular, it requested that progress being made in implementing these recommendations be reported to the Committee at its session in May 2001. It also underlined the importance of a greater specificity in evaluation recommendations, which should be targeted in terms of the responsible unit concerned, nature and timing of suggested actions. More broadly, the Committee highlighted the particular importance of introducing a set of criteria based on the priorities under the Strategic Framework in planning and selecting projects in future.
iv. Regarding the nature of future syntheses of project evaluations, the Committee agreed that these should focus on selected programmes and thematic topics in relation to the priorities of the Strategic Framework.
v. On the synthesis of thematic evaluations of TCP projects, the Committee appreciated that these exercises were undertaken with the initiative of the TC Department in order to enhance its ability to manage the TCP Programme, as well as in the interests of greater transparency and accountability. It considered that the synthesis brought out the strengths and weaknesses in TCP projects dealing with apiculture and sericulture on the one hand and legislative support on the other, and that it also pointed to a set of useful issues and lessons. However, some members requested that future reporting on TCP project evaluations include more details, including assessments of project performance by regions.
vi. The Committee generally endorsed the issues and related recommendations. In particular:
(Report of the 85th Session, May 2001 9 )
vii. The Committee welcomed this progress report in response to its earlier request. It took note of the measures aimed at improving the formulation and appraisal process, including the preparation of updated guidelines, a web-based formulation tool-kit training of staff in formulation techniques and strengthening existing mechanisms for project review and appraisal.
viii. The Committee recognized the importance of a clear division of responsibilities over the various phases of the project cycle, and looked forward to the contribution of a new service within the Field Operations Division in monitoring the project cycle, preparing guidelines and procedures and ensuring quality of project documents. The reduced operational units that would remain in the Regional Offices, would also have a key role, including review and operational clearance of project documents, whereas the in-country appraisal by FAORs of all new projects would be made more rigorous.
ix. The Committee agreed that it would need to return to this question at a future session once the appropriate mechanisms were in place and the reorganization of the TC Department completed.
x. The Committee considered that improving the quality of project design was important in enhancing the Organization's competitiveness for declining technical cooperation resources as well as its role in providing support to member countries in meeting their agriculture development needs.
6. While the results for the Near East and Europe are considerably below those for other regions, they are based on a much lower sample of projects.
7. In addition to the two TCP thematic evaluations summarised here, the analysis draws also on findings from the 1997 evaluation of food quality control projects.
8. PC 83/REP, paras 35-40.
9. PC 85/REP, paras 48-51.
