|PC 82/4 |
Rome, 13-17 September 1999
Evaluation in the Context of the Strategic Framework and the New Programme Model
1. When the Conference decided in 1997 to proceed with the introduction of a revised process for planning, programming and budget as outlined in the document JM 97/1 and called for a Strategic Framework to be prepared for FAO, it was foreseen that the evaluation regime in the Organization would also need to change significantly. The Programme Committee thus requested that proposals be made by the Secretariat on the nature and scope of a revised evaluation regime in the context of the Strategic Framework and the new programme model. This paper responds to that request. The proposals outlined below are put forward for discussion by the Programme Committee on the shape of the future evaluation approach and the reporting required by the Governing Bodies.
2. FAO was one of the earliest among the UN agencies to institutionalize evaluation, with an evaluation unit established in 1968, initially for evaluation of field projects. In 1979, the mandate of the Evaluation Service was extended to cover both the Field and Regular Programmes. Evaluation results have systematically been reported to the Governing Bodies, initially through separate reviews of the Field and Regular Programmes and since 1993 through the Programme Evaluation Report (PER).
3. Today evaluation is one part of the overall oversight regime in FAO and is located in the Office of Programme, Budget and Evaluation. The other components of the oversight regime are external audit, internal audit, inspection and investigation. While the External Auditor, as appointed by the Council, is responsible for External Audit, the remaining oversight functions are the responsibility of the Inspector-General with whom the Evaluation Service works in close cooperation.
4. Evaluation is both a management and an accountability tool. It provides managers at all levels with an improved knowledge base and critical assessment for their programmes. Its main purposes are (i) to catalyse improvements in overall planning and in the selection and design of programmes with respect to their usefulness, efficiency, effectiveness and impact; (ii) to support management decision-making for in-course corrections and improved execution; (iii) to provide input to decisions concerning the continuation of programmes at the end of their implementation period; (iv) to promote organizational learning on critical issues; and (v) to contribute to an overall increase in management accountability and in transparency of reporting to the Governing Bodies and other stakeholders.
5. Quality evaluation, which responds to the concerns expressed by management and the Governing Bodies, must be focused on results in terms of benefits to target users of FAO's services, rigorous in analysis, independent in assessment, consultative in process and transparent in reporting. Evaluation thus entails a critical analysis and assessment of performance in the achievements of the Organization, its programmes and projects. The criteria for assessing programmes include:
On the basis of this analysis, important issues and lessons are drawn for the future.
6. Techniques employed in evaluation include analysis against specific criteria as elaborated above; comparative benchmarking against similar programmes; and various methods of consultation with and feed-back from stakeholders, including questionnaires, sample surveys of results and focus group and key informant interviews. In all cases successful evaluation is based on informed and expert judgement and the application of rigorous means-ends analysis which links cause and effect.
7. The new programme model is intended to achieve a more strategic and coherent structure for FAO programmes. Within the Strategic Framework, the six-year Medium-term Plan (MTP) will provide a rolling plan to be updated biennially. This should contain a result-oriented, consolidated plan for all FAO programmes, with time-bound achievement targets and indicators for the medium term in pursuit of the objectives in the Strategic Framework. The programmes will be elaborated in terms of individual programme entities: technical projects, continuing programme activities and technical service agreements. The Programme of Work and Budget (PWB) will present the biennial workplan for programme implementation and related resources. As a result, programme execution is expected to clearly focus on results to be achieved in line with the Strategic Framework and the MTP which will entail rigorous planning, systematic monitoring and review for in-course correction and biennial updating.
8. Within the general definition of evaluation given in paragraph 4 above, it will be essential in the next few years to establish an appropriate evaluation regime that supports management in making an effective transition to the new programme model.
9. In order to assess overall progress towards Strategic Framework objectives, the new evaluation regime will need to examine the results of programmes, or clusters of programme entities, contributing towards those objectives as well as the performance of individual programme entities. It will also need to assess implementation of the strategies to address cross-organizational issues. The main frame of reference will be the MTP, using a number of different approaches:
10. The approach implies a need for evaluation to become more comprehensive and systematic in coverage, involving all FAO programmes, and requires the strengthening of the organization-wide system for monitoring, review and evaluation. However, not all programmes can be evaluated at the one time; rather, the evaluation cycle will tend to follow the rolling six-year medium-term planning cycle and, in particular, concentrate on completed programme entities or those nearing completion. In particular, comprehensive coverage for monitoring, review and self-appraisal at the divisional and departmental levels would have to play a critical role. The following will need to be put in place:
11. Continuing evolution and improvement will need to occur if the evaluation regime is to be successfully strengthened and this will particularly be the case for the coming transitional years, until the new programming approach is firmly established. As not all the areas for improvement can be achieved within existing resources, dialogue is sought with the Programme Committee on areas of priority. The main components of an optimal regime could comprise:
12. As is apparent from the components of the new evaluation regime described above, several modalities are envisaged:
13. The regime outlined above, if accepted, would have to be developed and refined over the coming years, and be underpinned by significant changes in the process of programme planning and management, as well as by adequate resources. Methodological problems would need to be overcome and a balance maintained between the desire for more and better evaluation and the cost-effectiveness of evaluation's contribution to greater impact from the Organization. Issues include:
However, the totality of the achievement of all the improvements indicated cannot be achieved within existing resources. In particular, the increase in the external input to evaluation called for by some members would necessitate substantial increases in resources. In view of this resource constraint, the Committee's views will be important on the balance to be given in evaluation work between:
- putting in place a stronger basis for evaluation, including firm establishment of the new programme model and establishing a functional monitoring and review process (auto-evaluation);
- ensuring effective feedback to users from evaluation and the wide availability of evaluation results; and
- the conduct of evaluation studies, in particular for accountability reporting to the Governing Bodies.
14. Reporting on evaluation, as part of accountability reporting from the secretariat, provides an in-depth, analytical basis for Governing Body decisions on programmes. Thus, reporting should cater to the needs of the Governing Bodies in the biennial planning and budget cycle, in which the Programme Committee and the Council, with advice also of the technical committees, perform the key preparatory work for final decision by the Conference. To serve the needs of the Governing Bodies, suggestions are made below as to: (i) the level of information required by the Governing Bodies; (ii) the content and presentation of reports; and (iii) the timing of reporting.
15. Level of information: More detailed information is normally required at committee level than in the plenary. The Programme Committee and the technical committees of the Council have the first line of responsibility for programme review. It is the Programme Committee which has, in practice, become the primary target audience for the PER, devoting more time to its review than any other body. It would thus make good sense to target evaluation reporting clearly at the Programme Committee. At the same time, both the Council and Conference still need to receive a report on programme evaluation for accountability purposes and for oversight of the evaluation function. This could, however, be presented in a more synthetic form.
16. Coverage of Reporting to the Programme Committee: The Committee may wish to propose changes in the way it approaches programme review in the light of the Strategic Framework and the new programme model. Programmes selected for in-depth evaluation could, to the extent possible, concentrate on programmes falling within the Committee's review cycle.
17. In-depth evaluation reporting of FAO programmes continues to be selective, given both the limited resources available for evaluation and time available to the Programme Committee. There is a trade-off between depth of analysis, especially as regards impact assessment and breadth of coverage (see previous discussion of impact). The Committee would continue to be consulted on the selection of subjects to be covered by evaluations for the forthcoming biennium. In line with the discussion of the evaluation regime above, products to the Programme Committee could include:
18. The implementation period to be covered would vary according to the types of evaluation. In general, however, in the case of programme or thematic evaluation, a sufficiently long period (some three biennia) would be needed to permit assessment of effects and impact.
19. Given the current resources of the Evaluation Service, the maximum number of biennial evaluation reports to the Programme Committee cannot exceed a total of four to five in various combinations of the above. It is likely that where a review would require a very substantial investigative effort such as in "a) assessment of overall progress towards a selected strategic objective", the number of other topics covered would need to be reduced to two or three.
20. Report content and presentation: While each evaluation report could be provided to the Programme Committee in a summary form (5-8 pages), supporting working papers to the reports could be made available to interested members of the Committee for consultation. Programme Committee evaluation reports could also be posted on the internet for the information of all members. Although the presentation would vary between field programme, thematic and programme reports the current framework of analysis presented would be retained; that is against the criteria outlined in paragraph 5 of this document.
21. As well as the reports themselves, the Committee would receive, as at present, a management response to the findings and recommendations and, where the report was prepared primarily by the Evaluation Service without significant external independent input, the comments of external peer reviewers.
22. Evaluation reporting to the Governing Bodies: This could thus comprise the following:
· a four to five page summary of each evaluation;
· a one page summary of the external peer reviewers comments;
· extracts from management's response;
· sections of the Programme Committee report relating to evaluations;
· comments of technical committees of the Council on individual evaluations, as appropriate.
23. Timing of Reporting : From the above discussion it would follow that:
24. Transition to the new arrangements: As evaluation reporting will not be able to reflect the results of the new programming arrangements and the approved Strategic Framework until at least two biennia of work have been completed, i.e. in 2004, the Programme Committee may wish to consider a reduced intensity of evaluation reporting in the interim period. This is particularly the case as the Evaluation Service is being called upon to support introduction of the improved programme model and ensure that it provides a future base for both organizational learning and accountability. In the meantime evaluation reports would include a mix of evaluations by technical discipline and theme. For the forthcoming biennium 2000-2001, it is suggested that the Programme Committee would review evaluation reports at its spring and autumn sessions which would then form the basis for the 2001 PER. Possible subjects are presented to the Programme Committee in a separate paper.
25. Programme Implementation Report (PIR) - In addition to the above proposed evaluation reporting, a biennial PIR on the previous biennium could continue to be produced. Given the inherently selective nature of evaluation reporting which is focused on effects and impact, the PIR would need to maintain its comprehensive coverage so as to form an integral part of accountability reporting. However, in the light of the new approach to programming and in view of some reservations expressed in the Governing Bodies on the content of the PIR, further efforts would be made to evolve a more satisfactory format for this report in consultation with the Programme Committee. As the new format of this report cannot be implemented until the completion of the first full cycle (i.e. 2006 at the earliest), the Secretariat intends to put further proposals on this subject to the Committee at a later date.
26. This paper is submitted to the Committee as part of an ongoing process of dialogue. To facilitate this dialogue, certain of the issues identified in the paper have been highlighted below so as to obtain the Committee's views on:
1 At its Eightieth Session in September 1998, the Programme Committee took note of "the move towards smaller-size and shorter-duration projects than in the past and the consequent need to develop appropriate cost-effective evaluation methodologies" (paragraph 34 of CL 115/8). Annex 1 addresses this issue.
1. At its Eightieth Session in September 1998, the Programme Committee took note of "the move towards smaller-size and shorter- duration projects than in the past and the consequent need to develop appropriate cost-effective evaluation methodologies" (paragraph 34 of CL 115/8).
2. In principle, all Trust Fund and UNDP projects with a project budget in excess of US$ 1 million are subject to formal evaluation. However, as is generally accepted, individual evaluations are not a cost-effective solution for small projects under US$ 1 million. Costs could easily exceed 5 percent of the project value and for very small projects of say US$ 100 000, a standard evaluation as applied to larger projects, even if somewhat scaled down, could cost 15 percent of the project budget.
3. It is difficult to classify small projects (budget less than US$ 1 million). By source of funding for 1996-97, the results by number of projects were: 45 percent FAO-TCP; 40 percent trust funds (of which 8 percent UTF) 14 percent UN funds (principally UNDP) and 1 percent other.
4. Since the PWB 1996-97, selected TCP projects have been evaluated on a thematic basis covering both ongoing and completed projects. The normal sequence is:
5. Both the preliminary desk studies and the field missions utilize standard check lists to ensure all major points are covered (see paragraph 5 in the main document). Field missions give particular attention to assessing the use made of the project outputs and thus their effect and impact. The inclusion of a considerable number of completed projects facilitates this analysis.
6. The three TCP thematic evaluations initiated to-date review some 80 projects of which 65 are subject to field visits. This represents about 5 percent of the total TCP portfolio in the period 1992-98, a proportion which will steadily increase as more thematic evaluations are undertaken. Funding for the evaluation work has been facilitated by the inclusion, in the standard TCP project document and budget of a provision of US$ 500 as a contribution towards the cost of evaluation.
7. In UNDP small projects, the FAO work generally constitutes an input to a larger project (this is also often the case for unilateral trust funds). Technical inputs to larger projects are normally reviewed in the context of the evaluation of the full project. FAO seeks to ensure that terms of reference cover assessment of the appropriateness, quality and results from the technical inputs and to be represented as much as possible on evaluation missions, so as to maximize feed-back. Where FAO is represented on the missions, the standard evaluation questionnaire is completed. This mechanism has proved adequate to date but the extent of project coverage by evaluation missions will be kept under review, as funding agencies may not always appreciate the need for FAO involvement in evaluation or the value of specifically reviewing the technical inputs.
8. Among small trust fund projects, emergency projects accounted for 13 percent and normative related projects for 27 percent by number. The remaining projects were a mix of activities but policy inputs, pilot and feasibility studies were significant. In pilot and feasibility studies, in-depth review forms an essential part of the process of formulating and appraising for possible follow-up. Also, the normative projects tend to be evaluated as part of the evaluation regime for the Regular Programme which is the subject of the main text of this paper.
9. It is further suggested that all small trust fund projects, including emergencies should be subject to thematic evaluation in the same way as is being accomplished for FAO-TCP and that a small direct charge should be included in the budget as standard practice.