1. The Programme Evaluation Report provides the Governing Bodies and senior management of FAO with an in-depth evaluative assessment of selected programmes and activities of the Organization. While by its very nature each report covers a small number of selected subjects, the intent is to cover a cross-section of FAO activities; substantive programmes (two or three programmes/sub-programmes), field operations (one topic) and organization-wide thematic topic (one). This fourth report covers two programmes (Programmes 2.1.4, Agricultural Support Systems, and 2.4.1, Forest Resources) in Chapters 1 and 2, one Major Programme 3.2, Support to Investment, in Chapter 3, a thematic evaluation of TCP projects on food quality control in Chapter 4, and a thematic review of participatory approaches to development in Chapter 5. As was the case for the last report, these topics were selected in consultation with the Programme Committee. It had also been intended to include a chapter on Programme 2.2.4 (Food and Agricultural Policy), but limitations on the size of the document prevented this from being included on this occasion.
2. The Conference, the Council and the Programme Committee have underlined the importance of evaluation both as a means of accountability and as a tool for learning lessons for future improvement. They have emphasized the importance of analytical focus in assessing results and impact as well as that of effective feedback from evaluation to the planning and implementation of programmes and projects. In reviewing the last report, they welcomed improvements made in its analytical quality and transparency. These include some new practices introduced, i.e. a) inclusion of programme managers' comments; b) the use of summary assessment based on a set of evaluation criteria; and c) inputs of external expertise in some of the programme reviews. At the same time, they reiterated their traditional concerns for enhanced use of evaluation in FAO, and encouraged further refinements and improvements, including "new" practices introduced in that report.
3. In the preparation of this report, these suggestions have been heeded to the extent practical. More intensive consultations were held with the programme managers and their senior staff in examining the findings, issues and recommendations arising from individual reviews, and their comments have been attached to each review in the report, including the reaction of senior management on broader issues. Efforts have been made to substantiate more explicitly the summary assessment given. Finally, on the use of external expertise two approaches have been used. Firstly, specialist consultants were engaged for the thematic review on participatory approaches and in the thematic evaluation of TCP projects on food quality control. Secondly, the review reports were commented by external peer reviewers. In the case of the Programme on Agricultural Support Systems, four specialists spent three days in Rome to review the draft report with the programme staff; for other programme reviews, comments were provided by correspondence. These comments are also attached to individual reviews in the report.
4. It is intended to continue with external peer reviews, which involved very modest direct cost. This pilot experience confirmed their usefulness in improving the content of reviews as well as their benefits to the programme managers concerned in providing a "second opinion" and in focusing attention on issues and recommendations arising from the process.
5. The reviews and evaluations contained in this report have been prepared by the Evaluation Service, which is responsible for the analysis and judgement therein. However, the Evaluation Service staff work closely with the programme managers and their staff throughout the process, which is valuable for informal feedback and mutual learning, over and beyond what is explicitly reported. At the same time, such internal reviews and evaluations have certain limits, particularly with respect to independence and expertise available. Thus, the use of external peer reviews has been employed, bearing in mind the need for cost-effectiveness and budgetary constraint.
6. As explained in the last issue of the Programme Evaluation Report, the programme reviews and evaluations presented follow the basic evaluation methodology as practised in FAO, which is largely derived from the logical framework approach and reflects a common methodology in the UN system, particularly for evaluation of technical assistance projects. This is mainly focused on assessing: relevance and coherence of the programme design, including objectives and strategic considerations; implementation efficiency in delivering the outputs and managing coordination among key participants; effectiveness in achieving the planned effects and impact; and sustainability of the results achieved. In order to facilitate reasonably in-depth analysis, the programme reviews cover a period of three to four biennia.
7. The approach used in the programme reviews presented in this report follows the structure shown below. This is the standard approach for selected programme reviews but it is suitably adapted to the requirement of the particular topic concerned.
8. While there is now a considerable acceptance of and interest in programme reviews and evaluations among the FAO programme managers and staff, there are certain important constraints in carrying out these exercises effectively. These include: a) weaknesses in programme design, including insufficiently clear objectives and achievement indicators beyond immediate outputs; b) the lack of systematic information on programme implementation achievement, particularly on results (how outputs and services have been used by the users and partners towards realizing the expected results); and c) time constraints on the programme staff, especially for managers, for adequate participation in the review process. Some of these constraints, particularly the first one, are being addressed through the strategic planning and improved programme planning processes.
9. The programme, operated by the Agricultural Support Systems Division (AGS), has provided a relevant and efficient response to developing country needs. Rural prosperity increases as a growing proportion of value-added occurs off-farm and farming itself becomes more of a business. The areas of work covered by the programme, i.e. Farming Systems Development, Agricultural Engineering, Post-harvest Management, Food and Agricultural Industries, Marketing and Rural Finance, have thus been particularly important for developing countries during the period of liberalization in their economies. The evaluation findings summarized below were broadly endorsed by an External Peer Review Panel which met in FAO Headquarters for three days.
10. Although there has been a lack of coherent strategy for the programme as a whole, this is now being corrected through an internal strategic planning process, which has identified specific cross-cutting areas for future concentration of effort. Individual sub-programmes have, in general, had a consistent purpose, with Food and Agricultural Industries and Post-harvest Management taking rather longer to adjust to a more normative role than other sub-programmes.
11. The programme has been generally efficient in producing outputs with the Marketing sub-programme being particularly productive. Some weaknesses occurred in this respect where there was insufficient focus on producing outputs for more general application in developing countries, which was to some extent the case with Food and Agricultural Industries. Quality outputs which may be highlighted include:
12. Although the overall quality of outputs was very satisfactory, areas identified for improvement included targeting of publications and ways of working in policy advice.
13. As with many programmes in FAO, internal joint work could have been greater but the programme's cost-effectiveness was leveraged substantially by strong external partnerships. These included joint work with the GTZ in Rural Finance and for the development of INPhO. The potential for joint work with UNIDO was not realized despite efforts by FAO. Important multiplier effects were achieved through networks and associations.
14. Information on the eventual impact of the programme's work was inadequate, although independent evaluation missions had almost uniformly reported positively on the results of field projects. Publication distribution had been limited, as is generally the case for FAO technical work but there was evidence that those referred to above were highly thought of. The networks and associations provided a framework for dissemination of results and the well-established, autonomous regional marketing and rural finance associations were themselves an impact of previous FAO work. It is generally agreed that FAO has been instrumental in carrying forward the concept of farming systems development from those of farming systems research. MicroBanker is a leading product in its field with evidence of substantial benefit to users. By 1998, it was installed at over 1 000 sites in 27 countries.
15. With growing reliance on the Regional Offices for direct support to countries, AGS at Headquarters will need to rapidly complete the reorientation move from Field Programme support towards meeting common requirements for practical policy instruments, approaches and information. The programme's effectiveness can now be further strengthened through greater internal integration. The mission for AGS is thus suggested as: "AGS will advocate and support the development of entrepreneurship in agriculture, agri-business and other agricultural support services in order to improve rural job opportunities and livelihoods and produce and market sufficient food and other agricultural products."
16. In addition to detailed comments on the future of work in individual subject matter areas, recommendations for the programme also address:
17. Issues raised for the Organization as a whole include:
18. The Forest Resources Programme, which is executed by the Forest Resources Division (FOR), is concerned with management, use and conservation of forest and tree resources. It covers ten programme areas ranging from the protection, assessment and use of these natural resources, particularly within fragile ecosystems and arid lands, to support improved forest management and land use practices within the framework of sustainable development. The programme performs FAO's Task Manager role for two of UNCED's Agenda 21 Chapters (Combating Deforestation Ch. 11 and Sustainable Mountain Development Ch. 13) and serves as the secretariat for the FAO Panel of Experts on Forest Genetic Resources and the International Poplar Commission. It has also been responsible for the implementation of activities that led to the Convention on Desertification (under Ch. 12) and the forestry aspects of the Convention on Biological Diversity (under Ch. 15).
19. During the period under review (1992-98), the programme has focussed, at the international level, on supporting the ongoing CSD-led international processes on the development of criteria and indicators for sustainable forest management (SFM). This included assisting developing countries in effective participation in these processes by initiating regional processes, particularly in Africa and for countries in arid and semi-arid areas. At the national level, it continued to provide technical support in a wide range of areas, including forest resources assessment, sustainable forest and fragile ecosystems management, forest genetic resources, combating deforestation, agroforestry, watershed and wildlife management. Since 1992, some 380 projects totalling close to US$ 307 million have been executed in these areas.
20. Main achievements during the period include:
21. Clearly, the work of the programme has been very relevant to the international and national concerns on forestry issues, evidenced by the high level of extra-budgetary support that has been attracted. This has, however, led to heavy workload driven by the demand for services in a growing number of priority areas. As a consequence, FOR staff have been overstretched, with some adverse effects on the implementation of normative work, especially in strengthening coordination and synergy with related efforts at the departmental and FAO levels and in promoting more cross-sectoral approaches linking forest resources with agricultural and rural development issues.
22. The Forestry Department formulated its strategic plan in 1996-97 to sharpen the strategic objectives and priorities for the major programme as a whole, but objectives at the programme level have remained more implicit than explicit. Given the financial and human resources constraints, it would now be important to establish more explicitly strategic perspectives for setting priorities under the programme.
23. This review recommends that:
24. The Investment Centre promotes investment in agricultural and rural development in developing countries through catalytic support in the identification and preparation of investment programmes and projects, acting also as an independent honest broker, and providing impartial technical advice. The Centre operates under collaborative arrangements with 20 international financing institutions (IFIs) with the major partners being the World Bank (under the FAO/WB Cooperative Programme (CP)), IFAD, and the African and Asian Development Banks (under the Investment Support Programme (ISP)) in the agricultural sector. All collaboration agreements are based on a cost-sharing principle, with the CP with the WB being the oldest and largest.
25. The Centre (TCI Division) maintains a strong multilingual, multidisciplinary technical staff of close to 100 professionals. They are complemented by a substantial number of international consultants (between 40 and 50 person/years annually) as well as a number of national consultants. RP expenditures on TCI increased slightly from US$ 18.9 million in 1992-93 to US$ 19.7 million in 1996-97 accounting for about 40 percent of the total expenditure throughout the period.
26. TCI's work has changed significantly during the review period, regarding the types of services and the mode of work. While TCI used to complete the project preparation process on behalf of governments, much of its work now is in support of national teams engaged in project preparation and implementation. Its role has been shifting towards advisory, guiding and training from one of doing the work by itself. TCI missions for such work are becoming shorter in duration, with fewer members with specialized expertise. This new approach, especially promoted by the World Bank and IFAD, contributes to national ownership of programmes and projects and enhances local skills and institutional capacities. Thus, in 1998, the number of missions doubled that of 1992.
27. While the traditional TCI work (sector/sub-sector studies and project identification/preparation) have remained predominant, there has been a substantial increase in tasks such as supervision, project completion reviews and support to FAO's SPFS programme.
28. The major achievements during the period include:
29. Potential issues relate mainly to TCI's role as a neutral interlocutor between member countries (borrowers) and the IFIs (the lenders), and TCI's ability to tap FAO's institutional resources and deliver high-quality advice:
30. This is the first of the thematic evaluations of TCP projects which were proposed to the Twenty-eighth Session of the Conference as part of the Programme of Work and Budget 1996-97. These evaluations were intended to draw lessons for future design of similar projects, determine achievements at project level, improve cost-effectiveness and provide a basis for accountability reporting. The first thematic topic chosen was Food Quality Control, projects which are technically supported by the Food and Nutrition Division.
31. Twenty-two projects that had been approved between 1992-1996 were chosen for the evaluation. Nineteen of them were visited by missions consisting of external consultants and/or staff of the FAO Evaluation Service, while the other three were covered in desk reviews. For the purposes of analysis, the projects were divided into four categories 1. Regional differences were noted in the types of projects implemented, with African and poorer Asian contries requiring a broader spectrum of assistance, Latin American countries requesting specific assistance in defined technical areas and European countries requesting assistance in formulation of legislation.
32. The Food Quality Control projects were found to be relevant to country needs in nearly all cases and there was good reason to believe at the time of formulation that there would be further action on the part of recipient governments. There was some tendency towards overly uniform project design and a more explicit link between problems identified and the input/output mix would have been desirable. Projects dealing with specific food control or food standards issues tended to be better designed than those which addressed food quality control more generally.
33. Projects were generally implemented on time and within budget. Personnel was the major budget item in most projects. International consultants performed well, as evidenced by the solid follow-up given in most countries to consultants' recommendations. Laboratory equipment was a small item in project budgets but significant in some countries in terms of upgrading capacity to carry out basic investigations. Many study tours were valuable, but some for laboratory personnel were not. Regional training courses in one project were found not to be cost-effective. Technical backstopping was sound and in future ESN officers could serve as lead consultants in projects.
34. The evaluation examined the results of various project components:
35. The evaluation concluded that FAO had a considerable technical capacity to implement technical assistance projects in food quality control. While follow-up to technical recommendations was very good, there was only one example of donors funding a subsequent project after TCP assistance ended. The evaluation recommended consideration of a Trust Fund for assisting countries to adopt Codex standards in various ways, as a means of following up food quality control TCP projects.
36. Given evidence of some overlapping recommendations, it was suggested that some consultancies could be combined in certain projects. It was also recommended that some consultancy visits might take place at the same time, thus improving chances for better coordinated advice.
37. The evaluation recommended greater targeting of industry in food quality control projects and stressed the importance of follow-up to training. Greater discrimination was recommended in the use of study tours, particularly for laboratory personnel. The need for equipment in project budgets should emerge clearly from the initial assessment, with a stated purpose for the component.
38. The mission also recommended a review of the FAO/WHO Model Food Law and, if found necessary, the issuance of revised guidelines for the development of national food laws which are general enough to apply in any legal system.
39. FAO has had a long association with participatory approaches to development, dating back to the early 1970s. The 1979 World Conference on Agrarian Reform and Rural Development (WCARRD) provided a major impetus to programmes aimed at strengthening institutional and organizational capacities of disadvantaged people to participate collectively in development efforts. In recent years, a number of developments have emerged, bringing about increased interest in and trends toward participatory approaches in development. These include: government downsizing and decentralization, the emergence of a broader and more assertive range of civil society actors and NGOs, new information technologies and networking possibilities.
40. Participatory approaches to development have been widely used in FAO and there is general awareness of the potential advantages of participation. FAO's work on participatory approaches to extension has been particularly successful and has shown that a strategic use of participatory tools can bring good results in projects with sectoral objectives (e.g. Farmers Field School concept, distance learning courses). FAO has also made major contributions to the body of knowledge on participatory approaches to development (e.g. PPP programme, community forestry). FAO has made extensive use of Participatory Rural Appraisal (PRA), including in the preparation of investment projects. Nevertheless, one important finding of this review is that despite the large volume of activities in this field, there is serious paucity of systematic information on their results and impact achieved on the ground.
41. From FAO's experience, it appears that in order for participatory projects to be successful and sustainable, the following conditions are necessary:
42. This experience, along with recent trends mentioned above, suggest that the major areas for further work are:
43. Coordination among FAO units on their participatory activities tended to be informal, through information sharing within the PRA network and through occasional cross-sectoral meetings to discuss particular topic areas. However, it would be desirable to strengthen, both formally and informally, collaboration between technical sectors within FAO on the topic of participation, since many issues at the community level are cross-sectoral in nature. On one hand, the informal group on participation in food security and sustainable livelihoods, which met for the first time in November 1998, could serve to strengthen more systematic linkages between other divisions and SDA. Besides suggesting to strengthen the group, it would be important for SDA to reinforce its capacity to play a proactive role as the lead unit in this thematic area.
44. Thus, the following recommendations are made:
1 General food quality control (including legal component); general food quality control (without legal component); projects for upgrading laboratories; projects dealing with specific food control and food standards issues.
