1. At its Hundred and Twenty-seventh Session in November 2004, the Council agreed to launch an Independent External Evaluation of FAO (IEE). In its report, the Council stated¹: “The evaluation aims at strengthening and improving FAO, taking into consideration FAO’s performance in conducting its mandate. In doing this, the evaluation process represented FAO’s contribution to the overall efforts of the international community to strengthen the UN system through appropriate reform. The Evaluation would consider all aspects of FAO’s work, institutional structure and decision processes, including its role within the international system. It could also be a resource for the review of the Strategic Framework.

2. The Council thus decided to establish an Inter-Sessional Working Group (ISWG) to formulate proposals for the scope, conduct and institutional arrangements for the evaluation for consideration by the Council. The ISWG would consist of a core group of up to three members of each regional group and the coordinator of the Group of 77. All Member Nations would be entitled to participate in the ISWG.”

3. “The Council decided that the ISWG would prepare proposals for:

terms of reference for a committee, to be established under Article VI of the FAO Constitution, to oversee, on behalf of the Council, the entire evaluation process;
terms of reference of the Independent External Evaluation of FAO, including the scope; content; methodology; composition, expertise and criteria for selection of the evaluation team; cost estimates; reporting process; and timetable for completion of the evaluation; and
definition of an appropriate supporting role for the Secretariat.

4. The ISWG would present its proposals preferably to the Hundred and Twenty-eighth Session of the Council in June 2005, and in any case not later than the Hundred and Twenty-ninth Session in November 2005.

5. All phases of the evaluation process, commencing with the work of the ISWG, would be financed from extra-budgetary resources, in full accordance with the financial rules and regulations of the Organization.”

6. In welcoming the Progress Report of the ISWG at its Hundred and Twenty-eighth Session, the Council reaffirmed the importance it attached to the Independent External Evaluation. The Council “applauded the thorough, inclusive and transparent nature of the ISWG process and expressed their appreciation to the Chair of the ISWG and to the Secretariat for their contributions which had made this possible. Representatives of regional groups stressed their expectations for the positive contribution of the evaluation in further increasing FAO’s effectiveness in fulfilling its mandate. The Council further emphasised:

the importance of ensuring adequate and timely voluntary extra-budgetary funding for the evaluation;
the continuation of the open and transparent way of working in order to achieve an outcome, which can be approved by all members;
that the ISWG should provide comprehensive and precise recommendations to the Hundred and Twenty-ninth Session of the Council on all the items in its mandate to facilitate definitive decision-making by the Council on the evaluation and its early commencement following that decision.”
in addition to the tasks assigned to the ISWG at the Hundred and Twenty-seventh Session, the Hundred and Twenty-eighth Session of the Council requested the ISWG to “make initial arrangements to facilitate the rapid and careful selection of evaluators and any supporting experts required for the IEE (such as quality advisers)”.

WORK OF THE INTER-SESSIONAL WORKING GROUP (ISWG)

7. In line with the Council decision, the ISWG was convened for the first time on 14 January 2005 by the Chairman of the Council, and appointed Ambassador Flávio Perri of Brazil as its Chairperson. Mr. Willem Brakel of the USA was subsequently appointed as Vice-chair. While the ISWG has made all substantive decisions, members agreed to establish a small Bureau² to act as a clearing house to facilitate the ISWG’s work, making proposals to the ISWG for its consideration. The ISWG held a total of twelve meetings and a positive, collaborative and transparent working atmosphere was maintained throughout. Nine member countries provided funds to a multilateral trust fund to support the work of the ISWG³ and a further four pledged contributions⁴.

8. After receiving nominations from the regional groups, the ISWG appointed two independent external experts with wide experience in evaluation to advise it in developing the terms of reference for the IEE: Mr. Horst Breier (Germany), and Mr. Dunstan Spencer (Sierra Leone) - for summary curriculum vitae see CL 128/15. A rigorous process was followed supported by the experts, beginning with the discussion of an Approach Paper by the ISWG, followed by detailed work on the terms of reference for the IEE, terms of reference for a Committee to oversee the IEE on behalf of the Council, and definition of an appropriate supporting role for the Secretariat.

ISWG DELIVERABLES AS DEFINED BY THE COUNCIL

Terms of reference for a committee, to be established under Article VI of the FAO Constitution, to oversee, on behalf of the Council, the entire evaluation process:

9. Annex I to this report addresses the Governance of the IEE and Functions of the IEE Council Committee. It is envisaged that the Committee of the Council be established on the same format as the ISWG. It will ensure the start-up of the IEE and provide monitoring oversight supported by two independent quality assurance advisers who will advise the Committee as to whether quality standards are being maintained for the IEE and the evaluation team is fulfilling the terms of reference. The conduct of the evaluation and its findings and recommendations will be the full responsibility of the evaluation core team under the overall direction of the core team leader.

Terms of reference of the Independent External Evaluation of FAO, including the scope; content; methodology; composition, expertise and criteria for selection of the evaluation team; cost estimates; reporting process; and timetable for completion of the evaluation:

10. The proposed terms of reference for the IEE are attached as Annex II. The terms of reference envisage the IEE beginning early in 2006. The final report of the IEE is to be considered by the FAO Council in November 2007, together with the response of the Director-General, and possibly after preliminary consideration by subsidiary bodies of the Council. It is thus essential for the report of the IEE to be available in at least advanced draft by July 2007.

11. As stated in the objectives of the evaluation in the proposed terms of reference “In analysing past and present processes and activities of the Organization, the evaluation is expected to be forward-looking and to emphasize findings, conclusions and targeted recommendations that would allow the Membership, the Director-General and the Secretariat of the Organization to chart the way forward, to better meet the challenges of the future in an evolving global environment, including newly emerging needs of member countries, and to position FAO, based on its strengths and comparative advantages. Consequently, the evaluation has the potential of becoming a milestone for FAO, reinforcing its role in a reformed UN system and the emerging new multilateral architecture. It should help to strengthen the sense of unity and purpose among the membership of the Organization, and to make FAO fit for the twenty-first century and the challenges ahead.”

12. The objectives also recall that, “the Director-General has stated in his foreword to his current reform proposals⁵, "I seek neither to anticipate nor to pre-judge the outcome of other processes underway, most notably the Independent External Evaluation of the Organization which the Council is undertaking. In fact, I believe that the implementation of my proposals now will create a more favourable context for such an evaluation”. The evaluation will thus take place during a period of ongoing reform in the UN system and adjustment in FAO which may accelerate to the extent that the Conference mandates further changes following its consideration of the Director-General’s proposals for reform. The evaluation will examine the effectiveness of the Organization’s work and its existing strengths and weaknesses and, in formulating its findings and recommendations, relate them to the validity of, and need for, further adjustment in the process of the Organization’s reform and its areas of priority.

Definition of an appropriate supporting role for the Secretariat

13. The supporting and facilitating role to be played by the FAO Secretariat is defined in Annex III.

Initial arrangements to facilitate the rapid and careful selection of evaluators and any supporting experts required for the IEE (such as quality advisers)

14. The ISWG has developed a procedure and journal/newspaper and web based advertisements for expressions of interest by evaluation consultants, in particular for the core team, the quality assurance advisers and the evaluation operations administrator. In this manner, it intends to ensure the widest possible pool for selection of candidates of the highest calibre with due attention also to geographical and gender considerations: At the time of preparation of this report, these advertisements had been put into circulation with a deadline of 10 November for responses and a database of interested consultants was being established.

Adequate and timely voluntary extra-budgetary funding for the evaluation

15. The ISWG agreed to propose to the Council a realistic indicative budget for the evaluation (US$ 4.3 million) which is included as Annex IV to this report. It also noted the request of the Council to the FAO Secretariat to open a multilateral trust fund to facilitate the early deposit of voluntary funds to fully cover the costs of the IEE.

16. In this context the ISWG notes the conclusions of the previous sessions of the Council that “All phases of the evaluation process, commencing with the work of the ISWG, would be financed from extra-budgetary resources, in full accordance with the financial rules and regulations of the Organization.” and that “it would not be possible to conduct the IEE without sufficient guaranteed resources. Under Financial Regulation 6.7, the Organization cannot incur expenditures against extra-budgetary resources prior to receipt of funds or at least a legal guarantee that such funds will be paid to the Organization”.

IN CONCLUSION

17. The ISWG has provided a unique and collegiate experience of working together in FAO governance by member country representatives. In the course of the ISWG’s work, all members have become further convinced that a fully independent, impartial and professional IEE can make a unique contribution in building a more effective FAO. The ISWG thus suggests to the Council that it urge all members to contribute according to their means in making this evaluation a true product of the membership as a whole. It further recommends to the Council that it decide that the Independent External Evaluation (IEE) of FAO should be initiated as soon as possible, subject to availability of adequate initial funds and full assurance of adequate funds being available at all stages of the evaluation process in order to complete the IEE report for consideration by the Council and Conference in November 2007. Furthermore, the Council is requested to adopt as annexed to this report the provisions developed by the ISWG for:

a. Governance of the IEE and Functions of the IEE Council Committee (Annex I);
(the Council is also invited to appoint a Chairperson for the IEE Committee of the Council);
b. Terms of Reference for the IEE (Annex II);
c. The Role of the FAO Secretariat in the IEE (Annex III); and
d. The Indicative Budget of the IEE including the provisions for support costs (Annex IV).

ANNEX I: GOVERNANCE OF THE IEE AND FUNCTIONS OF THE IEE COUNCIL COMMITTEE

The IEE Council Committee

18. The IEE Council Committee, assisted by a Bureau, will provide overall oversight for the management and operation of the evaluation, including on financial matters and adherence to standards of quality and independence. It will ensure that the terms of reference are adhered to in a timely manner, with quality and independence of process and outputs and within budget. Drawing on the advice of the quality assurance advisers (see below), Committee comments on findings and recommendations will thus be restricted to quality assurance, i.e. that the findings and recommendations are analysis and evidence based.

19. The Council Committee’s functions will include:

a) make all initial decisions for setting up the IEE, including, finalizing agreement on the selection of quality advisers, core team and evaluation operations administrator (initiated previously by the ISWG);

b) provide budgetary oversight;

c) consider the proposals in the inception report prepared by the evaluation core team regarding further detailing of methodology, areas of work and timetable for the evaluation and decide upon these;

d) take decisions on behalf of the Council if any aspect of the evaluation requires a definitive intervention (e.g. budget shortfall, major delays and adherence to terms of reference), but not as regards the conduct of the evaluation;

e) receive periodic reports from the evaluation core team, including progress on main areas of work and the final IEE report in draft and final and:

(1) provide feedback to the evaluation team on the extent to which standards of quality, independence and timeliness are being met (drawing where necessary on the advice of the quality advisers, who will be required to provide quality assurance reports to the Committee – see below);

(2) report progress to the Council at each session;

(3) resolve any issues between the evaluation team and FAO in its role as budget holder and contractor; and

f) forward to the Council the IEE final report and a report of its own activities.

20. Drawing on the advice of the quality assurance advisers in their reports, the Council Committee will take any necessary decisions to ensure the independence and/or quality of the work by the evaluation core team. It will, however, have no authority and will seek in no way to change substantive analyses, findings or recommendations or to exclude areas of work from the evaluation which fall within the terms of reference.

21. The IEE Council Committee and its Bureau will be constituted on the same lines as the ISWG, with a membership of 21and 7 regional representatives, respectively (the Committee and Bureau will also be open-ended in that any FAO member may attend). The Chair of the Council Committee will be appointed by the Council in a personal capacity, not as representative of his/her country and will also serve as Chair of the Bureau. The Council Committee will appoint a vice-chair from among its membership who will also serve as vice-chair of the Bureau. Members of the Council Committee will be encouraged to be represented in the Committee by evaluation specialists. The Bureau will stand in the same relationship to the Council Committee as for the ISWG, i.e. it will undertake preparatory work for the ISWG but have no decision-making authority. Member countries in the Council Committee and its Bureau will be responsible for the full costs of their participation in all meetings as has been the case for the ISWG. The Committee’s mandate and work will be completed upon submission of the IEE final report to the Council.

22. Meetings of the Council Committee and its Bureau will normally be attended as observers by the evaluation core team, the quality assurance advisers and the evaluation operations administrator. Secretariat support to the Committee will be provided by the Evaluation Service which will also manage that part of the IEE budget devoted to the work of the Council Committee.

The FAO Council

23. The Council will receive the final report of the IEE together with the response of the Director-General. The Council may decide (if necessary, even during the course of the evaluation) on whether it would wish any organ(s) of the Council to comment on the process of the evaluation and available findings and recommendations and advise the Council, prior to, or after the discussion of the final evaluation report by the Council itself.

The Quality Assurance Advisers

24. The quality assurance advisers, of which there will be two, will serve as and when required. They will have a key role vis-à-vis the Committee of the Council in preparing independent reports, and providing assurance on the independence and quality of the evaluation work. They will be present at each meeting of the Council Committee to provide feedback to the Committee and provide the Committee with an independent source of evaluation expertise. In addition, the quality assurance advisers will be responsible for monitoring reports to the Committee on the status of the work of the evaluation, in line with the work plan.

25. They will also provide, when requested by the core team, independent advice on any substantive aspect of the evaluation and will provide them with feedback as quality assurors if they should consider the evaluation work not to be achieving the necessary standards of quality or independence.

•

ORGANIZATIONAL STRUCTURE FOR THE IEE

IEE Council Committee

│

IEE Council Committee Bureau

Functions of Council Committee / Bureau:

• Make all initial decisions for setting up the IEE, including, if necessary, agreement on the selection of quality advisers, core team and evaluation operations administrator if not done previously by the ISWG and FAO Council;

• Provide budgetary oversight;

• Agree with the inception report’s proposals regarding further detailing of methodology, areas of work and timetable for the evaluation;

• Take decisions on behalf of the Council if any aspect of the evaluation requires a definitive intervention (e.g. budget shortfall, major delays), but not as regards the conduct of the evaluation;

• Report progress to the Council at each session;

• Receive periodic reports of work and the final report of the IEE in draft and final; and

• Forward to the Council the IEE final report and a report of its own activities..

FAO Evaluation Service

Functions

• secretarial support to Council Committee;

• budget holder providing financial monitoring

• facilitate core team work including contacts with FAO Secretariat.

Quality Assurance Advisers

Functions:

- To Council Committee: provide independent reports on:

• quality of evaluation work and deliverables;

• independence of the evaluation;

• progress of activities and deliverables against schedule.

-To core team:

• provide, when requested, independent advice on any substantive aspect of the evaluation;

• advise on the quality of the work.

•

Evaluation Operations Administrator

Functions – Reporting to the core team leader; responsible for the administrative support to the evaluation, including:

• interface with budget holder;

• interface with FAO on support and logistics;

• recruitments;

• travel;

• contacts.

•

•

•

•

Core Team

Composition - core team leader, and team leaders for: governance; technical work of FAO; management and organization; and FAO’s role in the multilateral system (total 5 people).

Functions – Responsible for all substantive matters of the evaluation, including full involvement in the conduct of the evaluation:

• Decisions, within the agreed ToRs, on methodology for the evaluation;

• Selection of supporting evaluators and research assistants;

• Leadership of supporting evaluators and research assistants, including specialist teams;

• Ensuring coherence of the evaluation;

• Preparation of periodic reports, reports on each main area of work and the final report;

Finalization of, and final decision on, all reports, including the findings and recommendations

ANNEX II: TERMS OF REFERENCE - INDEPENDENT EXTERNAL EVALUATION (IEE) OF FAO

I. Background

26. FAO was founded with a membership of 42 countries in October 1945 to ensure humanity’s freedom from hunger through the promotion of agricultural development and trade, improved nutrition, rural development and the pursuit of food security. Today, the Organization serves 188 Member Nations and one Member Organization, the European Community.

27. Since 1994, FAO has been engaged in a programme of reform and has made efforts to overcome organizational weaknesses and to deliver its services more effectively. A plan to refocus, reorganize and reinvigorate the Organization was enacted with the approval of the Governing Bodies. Key elements of this plan included restructuring of the Organization’s functions to provide greater focus for normative and operational work and to achieve synergies between the two; downsizing of personnel; equitable representation of member countries; achieving gender parity; streamlining of processes and procedures; decentralization and assignment of more responsibilities to staff in the field; modernization through the use of new technologies; forging new partnerships; strengthening the cooperation with donor countries; and a communication strategy.

28. In November 1999, the FAO Conference at its Thirtieth Session approved the Strategic Framework for FAO 2000-2015. This framework defines a set of strategies that are based on the principles of interdisciplinary work and partnership and are designed to provide a platform in the drive to ensure that the Organization meets new challenges in a changing world.

29. At the same time, international developments in recent years have become increasingly dynamic. This is marked, among others, by: the Millennium Declaration adopted by 189 Heads of States and Governments at the UN in the year 2000, including the eight Millennium Development Goals; the World Food Summit Goal; the reform of the UN system; the emergence of a new international architecture in the area of FAO’s competence and mandate; and major international events such as the Monterrey and Johannesburg Conferences⁶ and the Doha Round of Trade Negotiations; and, most recently, the UN Summit Declaration of September 2005. In September 2005, the FAO Director-General presented a further set of far reaching reform proposals for consideration by the Conference in November.

30. There also have been fundamental changes in national approaches to development and international cooperation, characterized by ownership and partnership and by harmonization and alignment for greater aid effectiveness. International developments are characterized by the continuing threat of international terrorism and the fight against it; by natural and man-made disasters leading to humanitarian catastrophes; and by a much more differentiated assessment of the costs and benefits of globalization and a growing uneasiness in many parts of the world with the negative impacts of globalization on significant groups of people.

31. Progress has been achieved in the overall performance of the Organization. However, a comprehensive, integrated and in-depth analysis of where FAO stands today is now essential, especially in the context of a dynamic and rapidly changing world under globalized conditions. The evaluation will take place during a period of ongoing reform in the UN system and in FAO which may accelerate if the Conference mandates further changes following the recent presentation by the Director-General of reform proposals for its consideration.

32. At its Hundred and Twenty-seventh Session in 2004, the FAO Council agreed to launch a comprehensive Independent External Evaluation of FAO. The Council also decided to establish an Inter-Sessional Working Group (ISWG) to formulate proposals for the evaluation for consideration by the Council, including the formulation of proposed terms of reference. These terms of reference (ToRs) for the Independent External Evaluation of FAO are based on the consensus-building processes and the decisions in the ISWG and draw heavily on the expert input by two independent consultants to the ISWG (Mr. Horst Breier – Germany and Dr. Dunstan Spencer – Sierra Leone).

II. Objectives of the Evaluation

33. When approving the launching of the Independent External Evaluation of FAO, the Council agreed that “the evaluation aims at strengthening and improving FAO, taking into consideration FAO’s performance in conducting its mandate. In doing this, the evaluation process represented FAO’s contribution to the overall efforts of the international community to strengthen the UN system through appropriate reform. The evaluation would consider all aspects of FAO’s work, institutional structure and decision processes, including its role within the international system. It could also be a resource for the review of the Strategic Framework.”⁷

34. Thus, the Independent External Evaluation of FAO will be comprehensive in scope, reflecting the aspirations and concerns of the whole membership. Findings, conclusions and recommendations emanating from this evaluation will be directed to the Governing Bodies of the Organization, to the Director-General and to Member Nations for their review and action. They will also contribute to informing the policy debate of Member Governments as well as that of the UN and the wider international system. Similarly, the evaluation will help to convey to the general public an evidence-based assessment of the achievements of the work of FAO and of the challenges lying ahead for the Organization.

35. The Director-General has stated in his foreword to his current reform proposals⁸, "I seek neither to anticipate nor to pre-judge the outcome of other processes underway, most notably the Independent External Evaluation of the Organization which the Council is undertaking. In fact, I believe that the implementation of my proposals now will create a more favourable context for such an evaluation”. The evaluation will thus take place during a period of ongoing reform in the UN system and adjustment in FAO which may accelerate to the extent that the Conference mandates further changes following its consideration of the Director-General’s proposals for reform. The evaluation will thus examine the effectiveness of the Organization’s work and its existing strengths and weaknesses and, in formulating its findings and recommendations, relate them to the validity of, and needs for, further adjustment in the process of the Organization’s reform and its areas of priority.

36. The analysis will be approached from the point of view of the crucial needs and concerns in a situation of limited resources where not all expectations can be met. In assessing this, full account will be taken of the views of member countries on the services they require and receive, on their quality, effectiveness and impact, and on their relevance. Comparison with arrangements in other agencies will provide a useful bench-mark.

37. In analysing past and present processes and activities of the Organization, the evaluation is expected to be forward-looking and to emphasize findings, conclusions and targeted recommendations that would allow the Membership, the Director-General and the Secretariat of the Organization to chart the way forward, to better meet the challenges of the future in an evolving global environment, including newly emerging needs of member countries, and to position FAO, based on its strengths and comparative advantages. Consequently, the evaluation has the potential of becoming a milestone for FAO, reinforcing its role in a reformed UN system and the emerging new multilateral architecture. It should help to strengthen the sense of unity and purpose among the membership of the Organization, and to make FAO fit for the twenty-first century and the challenges ahead.

38. The cornerstones for the evaluation of FAO have been clearly defined by the Council. The evaluation is to be comprehensive, external, independent and professional. It will cover the institutional performance as well as the technical work of FAO, operational as well as normative. The focus will be on relevance, efficiency and intended and unintended results, including outcomes and impacts and their effectiveness and sustainability.

39. The Mandate of FAO, as laid down in the Preamble and Article I of the FAO Constitution, is taken as the basis and will not be open to question by the evaluation. The mandate will provide the overarching yardstick for assessing the performance and the impact of the work of the Organization, which plays an important role as a multilateral institution, with equality between members, providing knowledge-based global public goods and development services. Other important sources of reference in examining the Organization’s work include contributions to the goals and objectives established by the World Food Summit, the Millennium Declaration, the Conferences of Monterrey, Johannesburg and Doha and the UN Summit Declaration of 2005. The Organization’s own Strategic Framework approved by the Conference in 1999 (immediately prior to the Millennium Declaration) and the subsequent Medium-Term Plans will also provide important statements against which to gauge the Organization’s performance.

40. In order to meet the requirement of comprehensiveness the evaluation will encompass the following four key components of analysis, which are all closely interlinked:

Technical work of FAO: There is a vast array of technical work of FAO that has to be evaluated and analysed with regard to its relevance, efficiency, outcomes, impacts and sustainability. The evaluation will examine the main elements of the technical work of the Organization, including the interconnectedness between them. These elements include work addressed to overcoming hunger, safeguarding the environment and improving the conditions for economic and social development, including gender mainstreaming and attention to the rights of children. Issues are addressed through: advocacy; policy development and advice; regulatory and standard setting work; information; statistics; studies; technical cooperation; emergency response; networking and dialogue with respect to: access to food, crops, livestock, forests, fisheries, commodity trade and rural development. It includes policy dialogue and agreement in international meetings, including those of the FAO Statutory Bodies. The technical work of FAO also includes lead programmes, such as the Special Programme for Food Security and encompasses the total effort in building global, regional and national knowledge and capacity. The analysis will cover issues of relevance and timeliness; the Organization’s service orientation responsiveness to expressed and changing needs of member countries; supply-side and demand-driven elements in shaping FAO’s work; the number of activities versus the depth of their treatment; project versus programme approaches; the matching of work programme and resources and their impacts.
Management and organization of FAO: This includes budget, administrative and financial systems; the organizational structure of FAO (e.g. departmental set-up; decentralized structures; accommodation of cross-cutting issues; oversight; and evaluation); the Organization’s corporate culture, including the existence of enabling environments for the full utilization of staff potential, the delegation of authority and the requirements of a knowledge organization; risk management; the human resources policy and management including accountability for gender mainstreaming; the decentralization and the respective roles of headquarters, regional and country offices and the opportunities and constraints of this arrangement; communication strategies; as well as infrastructure issues.
FAO governance: This includes the roles and the efficiency and effectiveness of the Governing Bodies (Conference, Council, Programme and Finance Committees, and Committees on Agriculture, Commodity Problems, Fisheries, Forestry and World Food Security); key aspects of the relationship between the members and the Secretariat, as exemplified, for instance, in the area of priority setting and in programming and budgeting processes; the funding structure of FAO (regular budget and voluntary contributions), including the opportunities and constraints of the present structure and the extent to which it contributes to member ownership of the Organization and its multilateral character; governance relationships within the UN system; and wider participation of stakeholder groups.
FAO’s role in the multilateral system: The evaluation will examine the role of FAO within the multilateral system based on the demonstrated strengths of the Organization, its comparative advantages and its ability to enter into alliances and contribute to the UN and wider international system as a whole. Important issues for the evaluation findings and recommendations thus include: critical gaps in the international architecture in FAO’s area of mandate; identification of those areas where the mandate is well filled by a number of other agencies; and the questions of comparative advantage, and the related issue of bench-marking against other agencies. Partnerships, including those with non-UN actors such as international agricultural research institutions, regional organizations, international NGOs and business organizations, and in particular partnership with the Rome-based UN agencies, will all be areas for enquiry. These will be evaluated in conjunction with the three preceding components so as to link the role of FAO in the multilateral system with concrete examples of work such as that on international trade and international efforts to achieve the Millennium Development Goals (MDGs).

41. Utmost care will be taken to maintain the comprehensiveness requested by the Council throughout the evaluation process, to secure a holistic approach to the evaluation and to assure that synergies are explored and fully developed and that the interconnectedness of the different components of FAO processes and technical work are adequately reflected in the evaluation. The core team (see below) will have responsibility for this task.

42. Although the evaluation will be comprehensive, the evaluation team will have the independence and degree of flexibility, within the scope of the ToRs, to define and concentrate on those areas in which it feels there are particular strengths to be built and weaknesses to be addressed, and to explore in greater depth those issues which it identifies as being of importance. The team will ensure, however, that this process will be free from any biases that could undermine the independence, impartiality and credibility of the evaluation, and that it has the expertise and time to deal with the issues selected.

III. The Evaluation Team and its Role

43. The core team will consist of four members, including the team leader. Under the direct authority of the team leader, the core team will have the sole responsibility for the direction, supervision and conduct of all substantive work of the IEE, including full involvement in the execution of the evaluation work. Each member of the core team, including the team leader, will have responsibility for the leadership of one of the four main areas to be covered by the evaluation as discussed above (para. 40). The core team functions will include:

decisions on methodology and approach, including defining the work plan, within the Council agreed terms of reference, budget and timeframe;
selection of countries for country visits and FAO programmes for case studies, according to the criteria set out in the terms of reference and confirmed in the inception report;
selection of supporting specialist evaluators and research assistants in open competition;
leadership of the evaluation including leadership of specialist teams and supervision of all supporting evaluators and research assistants;
ensuring coherence of the evaluation;
preparation of periodic reports, reports on each main area of work and the synthesis report; and
finalization of, and final decision on, all reports, including the findings and recommendations.

44. Specialist evaluators: The core team will be supported in its work by specialists, who will bring in their knowledge and experience with the international intergovernmental system as well as additional cutting edge knowledge and experience (for example from the practice of the private and NGO sectors and from academia).

45. Evaluation teams: Interdisciplinary teams including, where possible, a member of the core team, will undertake visits to countries, decentralized FAO offices and to other organizations of the multilateral system. In as far as possible, these teams will have common membership, i.e. consultant evaluators will take part in teams visiting several parts of the world. These same consultant evaluators will also in most cases be members of the specialist teams evaluating FAO’s technical programmes and, where appropriate, working on management and organization, governance and FAO’s role in the multilateral system. The supporting specialist evaluators in each area of work will therefore be limited in number and take part in several field missions, plus the headquarters work.

46. Annex II Appendix 1 provides information on the competencies required for the core team and supporting specialist evaluators and the criteria for their selection. The evaluation core team will be supported in its work by an evaluations operations administrator (see also Annex II Appendix 1).

47. The evaluation team will be overseen in its work by the Committee of the Council for the IEE. This Committee will provide overall oversight for the management and operation of the evaluation, including on financial matters and adherence to standards of quality and independence. It will be responsible for approving the proposals of the core team for the conduct of the evaluation in the inception report (see Annex I, Governance of the IEE and Functions of the IEE Council Committee).

IV. Scope of the Evaluation

Key evaluation issues

48. Underlying the approach to all aspects to be covered by the evaluation will be the fundamental questions common to evaluations, which include:

key changes in the external environment in which FAO functions;
FAO’s relevance to the needs and priorities of the governments and people of member countries and the international community;
functionality and clarity of the objectives, strategy, design and implementation plan to meet those needs and priorities;
efficiency and effectiveness of the processes followed;
institutional strengths and weaknesses, including institutional culture and the inclusiveness of process;
quality and quantity of outputs, in relation to resources deployed in undertaking the work;
quality and quantity of the outcomes (effects) resulting from the activities and outputs also in relation to resources deployed for the work;
impacts and their sustainability in terms of benefits to present and future generations for food security, nutrition, social and economic well-being, the environment, etc.; and
FAO’s comparative advantage in addressing the priority needs.

49. There are a number of overarching or lead evaluation questions which deal with impact; needs and priorities; comparative advantage (including gaps in the international architecture); and efficiency. They need to be answered in order to arrive at an overall assessment of the impact of FAO and its work. These questions inter alia include:

a) FAO’s overall institutional strengths, weaknesses and contribution to sustainable impacts, (including the issues of gender equity and rights of children): - in addressing for the Organization’s areas of mandate:

i) The Millennium Development Goals, the World Food Summit Goal, and the goals of FAO Member Nations as agreed in the FAO Strategic Framework, in particular those relating to:

1) hunger and poverty (MDG 1) - To what extent does FAO contribute to the eradication of food insecurity and rural poverty? and

2) the Environment (MDG 7) What are the results of FAO’s support to the conservation, improvement and sustainable use of natural resources for food and agriculture?

ii) Rural and national socio-economic growth and development, in particular as it relates to poverty reduction. In what way does FAO contribute to creating sustainable increases in the supply and availability of food and other products from the crop, livestock, fisheries and forestry sectors and what is the income generation from this?

iii) Global and regional requirements for information. Does the provision by FAO of information and assessments and its fostering of knowledge management for food and agriculture lead to improved decision-making nationally and internationally? and

iv) Global and regional requirements for international legislative, standard setting and regulatory frameworks. How successful is FAO in promoting, developing and reinforcing policy and enabling frameworks in standards and regulation for food, agriculture, fisheries and forestry?

b) To what extent do FAO resource levels and the application of limited resources reflect members’ priorities and needs and are they commensurate with the tasks the Organization is attempting to undertake, its comparative advantages, and areas of effectiveness as revealed by the evaluation analysis?

c) What is the changing global perspective for requirements in FAO’s area of mandate?

d) How does FAO focus in its area of mandate and what is its capacity to:

identify and adjust to changing needs and priorities; and
identify improvements and adjust its institutional structures and ways of working in line with changing needs and opportunities created by new technology, improved communication, etc.?

e) What is the relevance, potential for effectiveness and implementability of the Director-General’s reform proposals, including those agreed for implementation by the Conference (November 2005) and what is the continued relevance and adequacy of key FAO strategy documents, including the Strategic Framework?

f) To what extent in achieving the Organization’s objectives and goals are the following conducive: governance structure and practice; budget and funding arrangements including the relationship between the core regular budget and extra-budgetary resources; institutional structures; management culture; administrative and financial systems; and human resources policy and practices?

50. The IEE will also pay attention to the process through which FAO ensures the adoption and implementation of recommendations generated by its own evaluations and other oversight and quality assurance activities.

51. Annex II Appendix 2 provides a check list of issues identified by the ISWG for the reference of the core team in preparing its evaluation inception report.

Period covered by the evaluation

52. The evaluation is forward looking. Its central concern is thus in identifying strengths and weaknesses in FAO’s programmes, approaches and structures with relevance for the future. In examining the institution’s capacity to change, flexibly responding to medium-term changes in members’ requirements and the external environment, it will be necessary to examine the background of reform in the Organization since the present round of reforms was initiated in 1994. However, in examining the effectiveness and impact of programmes a time-frame examining the outcomes and impacts of work undertaken over the last four to six years will generally be appropriate, as for longer periods than that both detailed information and the lines of causality in terms of impact become difficult to trace. For many institutional issues, the evaluation will be essentially concerned with the efficiency and effectiveness of current, rather than historical, practice as well as the likely benefits of ongoing reforms.

V. Methodology of the Evaluation

53. It is expected that the evaluation will apply the Norms and Standards for Evaluation in the UN System, as approved by the United Nations Evaluation Group (UNEG) in April 2005 (these are largely in accordance with the OECD-DAC Principles for Evaluation). There are a number of standard elements of evaluation methodology that would need to be drawn on in any evaluation. They comprise well-tested social science methods for sampling; the identification of indicators; benchmarking; guidelines for interviews (open, structured or semi-structured; face-to-face, by telephone, or in group sessions); the use of questionnaires and their design; triangulation; validation and weighting. The range of methods available also includes simple tools for cost–benefit analysis; participatory data collection (such as rapid rural appraisal techniques); the design of an overall evaluation matrix; and stakeholder/verification and peer review workshops.

Maximising the use of existing information

54. The IEE is conceived as maximising the use of existing information. This will start with the preliminary review necessary to prepare the inception report and will be continued throughout the evaluation process. The core team will carry out a desk review of FAO strategy and corporate policy documents, evaluation reports, guidelines, country programmes and major outputs, tracing the course of development since the start of the current phase of reform in 1994 but concentrating on the last six years. This will be supported by a wide range of interviews with representatives of member countries and the FAO Secretariat during the inception phase.

55. The evaluation will maximise the use of existing evaluations and similar work and will thus not itself examine all aspects in detail but will, to the extent possible, rely on the work of others in forming its judgements. Several of the more recent independent evaluation reports produced by FAO’s own Evaluation Service are believed to provide useful information and not require duplication but the evaluators will need to assess the quality of this existing work, including its independence and impartiality. This should also include examining the extent to which the reports of previous evaluations have been acted upon by the Organization.

Assessing impacts

56. The evaluation team will have to rely for the most part on secondary data from FAO’s own evaluations and other documentation and from stakeholders in member countries in both verbal and written presentations. It will be imperative for the consultants, however, to assess on their own the quality of the data that they intend to use and where possible to back this up with some primary data, perhaps through separately commissioned country and/or programme impact studies. Areas for impact assessment will be carefully selected in the light of these constraints. In view of the relatively small inputs by FAO to development processes at the national and global level, key questions will concern the extent to which there has been contribution to a plausible line of causality. The inception report should include specific proposals for impact assessment.

A consultative and transparent process

57. Consultation with all stakeholders will be key, in order to ensure confidence and ownership in the evaluation process. During the inception phase it will be important in determining issues, areas for concentration, etc. It will also be essential for information gathering; to verify findings and to examine the potential implementability of recommendations (which is expected to be particularly important in the areas of governance and of administrative, financial and human resource procedures). In the countries and decentralized office site visits, as well as at headquarters, consultations and interviews with government representatives, civil society, the private sector, NGOs, development agencies, in-country coordination and advocacy groups, policy research bodies, and beneficiaries, will all be important. In addition, a number of stakeholder workshops may be considered in key areas such as governance and to verify major outputs of the evaluation. Questionnaires and possibly electronic bulletin boards will also be important in obtaining an input from all stakeholders, as well as helping to ensure transparency and ownership.

58. Major intermediate evaluation deliverables, such as the inception report will be made available on a public website dedicated to the IEE.

Samples for in-depth evaluation and analysis

59. Sampling will be informed by the review and consultation process in the inception phase, which may also usefully include a preliminary analysis of FAO strengths, weaknesses, opportunities and threats (SWOT).

60. Country visits and country case studies: It is essential for all aspects of the evaluation that the evaluation team visit member countries, in addition to working through other forms of enquiry such as questionnaires and telephone interviews. It is through country-level studies that results of FAO’s work will be confirmed and the views of member countries most fully explored. Countries to be visited should be selected by the core team on the basis of a set of clearly defined, transparent criteria which should be detailed in the inception report.

61. With a focus on those countries where there are large numbers of poor and hungry people, sampling will be purposive for selected groups of countries to ensure that key variables, including: regional balance; level of development; the number of malnourished; both large and small populations; and the size of the FAO programme (normative and technical and emergency assistance) are well represented. Logistic considerations will also be a factor. Within these criteria selection will be randomised (stratified random sampling).

62. It is envisaged that country visits will cover some 12-14 percent of FAO membership (about 25 countries), which would enable a representative sample of key countries to be covered. At the same time, it would be necessary to visit other international organizations and members of the UN system working in FAO’s area of mandate. To the extent possible, this could be combined with country visits. A limited number of more in-depth country case studies of the FAO programme as a whole, or particular aspects of it may also be envisaged by the evaluation team.

63. Teams making country visits will cover all areas of the evaluation including governance and organization and management issues. The plan for country visits and related contacts should be formulated by the core team in line with the criteria defined above, and presented to the Committee of the Council, preferably as part of the inception report.

64. Prior to undertaking country visits, a workshop could usefully be convened for all those who will be involved in field work to assure the commonality of understanding and approaches to the country work, including the methodologies applied and the comparability of findings from field work.

65. Programmes for in-depth evaluation: In addition to the overview of all programmes required for the comprehensiveness of the IEE, more detailed analysis will be important on a sample of programmes and cross-cutting issues. The core team will be expected to present its proposals for this in the inception report. One criterion in this selection will be the extent to which work has already been assessed by existing evaluations. Also, with the aim of forward looking evaluation, criteria may include: the size of the programme or area of work; the demand from member countries; and areas of work being considered for expansion because of their perceived relevance and usefulness; or for elimination or downsizing. It is also the case that when work is already being reduced because of general agreement on its lack of continued priority, evaluation can be useful for accountability but is not likely to deliver forward-looking lessons.

66. In-depth analysis of issues in management and organization (resources, budget, administration, finance and human resources): As with programme evaluation, an overview is required of the totality of the issues but in sampling those areas for more in-depth study and criteria will include the availability of existing information from audit reports, management consultancy, and internal studies. In addition, criteria may include, the extent of risk, both financial and political (public image) risk; perceived potential for efficiency gains; and the importance of the area for the Organization’s delivery.

Consideration of other institutions and benchmarking

67. The IEE will need to review the work of institutions other than FAO, especially in the multilateral system. This will be important for benchmarking on procedures, processes, quality of work, etc. As the performance of FAO cannot be judged in isolation from that of its partners and competitors, it will also be essential to make judgements on FAO’s areas of comparative strength and weakness in the multilateral system, with respect to its areas of mandate. The inception report should provide clarification on how, in addition to visits to other institutions, the evaluation intends to approach assessment in this area.

Recommendations of the IEE

68. The core team is solely responsible for the evaluation findings and recommendations but it is expected to consult widely on these in order to ensure both their factual evidence base and the potential for practical follow-up action. Where appropriate, alternatives may be presented with their advantages and disadvantages. Evaluation recommendations should, to the extent possible, be presented in operational terms, while respecting the roles of management and the Governing Bodies in developing operational plans.

VII. Deliverables and Timetable

69. Deadline for final report: The final report of the IEE is to be considered by the FAO Council in November 2007, together with the response of the Director-General, and possibly after preliminary consideration by subsidiary bodies of the Council. It is thus essential for the final report of the IEE to be available in at least advanced draft by July 2007.

70. Deliverables: The inception report will specify the key deliverables proposed for delivery by the IEE core team for consideration by the IEE Committee of the Council (see Annex I) The Council Committee is responsible for approving the inception report for the evaluation prepared by the core team and “it will ensure that the terms of reference are adhered to in a timely manner, with quality and independence of process and outputs and within budget“.....“comments on findings and recommendations will be restricted to quality assurance, i.e. that the findings and recommendations are analysis and evidence based.“ Deliverables can be expected to include, among others to be identified during the course of the evaluation work:

inception report (to be presented for approval by the IEE Committee of the Council some two months after evaluation start-up);
progress reports, including reports on individual areas covered by the IEE (technical work, governance, etc.); and
drafts and final of the report of the IEE for consideration by the FAO Council.

71. The inception report: The first task of the core team will be to prepare an inception report, within two months of evaluation start-up, for approval by the Council Committee for the IEE. In preparing its proposals in the inception report, the core team will take account of the considerations for coverage, issues and methodology discussed above. The core team is, however, encouraged to suggest alternatives where it considers this appropriate, as well as including additional considerations in its proposals. The inception report will provide a comprehensive road map for the evaluation, including proposals on:

issues to be addressed by the evaluation and how it intends to address the range of issues identified;
methodology proposed for evaluation and criteria for selection with respect to:

countries for visits and for case studies and the plan of visits and studies based on those criteria;
programmes and areas of work for more in-depth evaluation, and the programmes selected;
governance, institutional, administrative, financial and human resource issues for study in depth;
other agencies to be visited and studied for purposes of benchmarking and reviewing FAO’s place in the multilateral system; and

the plan of visits and studies based on those criteria.

72. All deliverables will be as concise as possible and submitted in English. The language used should be direct, free of jargon, avoid euphemisms in describing problems and weaknesses, and be reader-friendly. Annexes and appendices should only be included if there is a clear rationale for doing so. Executive summaries should be included and address findings and recommendations. If certain issues agreed for analysis in the inception report could not be addressed satisfactorily in the course of the evaluation, the reasons should be explained.

73. Website: Deliverables will be placed on a special web site to be created for this evaluation and thus be made accessible to a wider public in order to strengthen the transparency of the evaluation process.

Annex II Appendix 1: Required Qualifications for the Evaluation Core Team and Evaluation Operations Administrator

74. Candidates for all posts will be selected on the basis of technical competence. Regional and gender balance will also be considered. Language capability will be an important factor in selection, with good spoken and written English essential and knowledge of Arabic, Chinese, French and/or Spanish an advantage and a significant selection factor. In order to avoid conflicts of interest, for the core team persons who have been FAO staff members within the last three years, who have undertaken substantial non-evaluation work for FAO within the last three years, or who have represented their governments in the governance structures of FAO within the last three years, are excluded from consideration. Those in continued employment of their national government or an organization which works directly at international level in FAO’s area of mandate, are also excluded.

75. The core team, under the direct authority of the team leader, will have sole responsibility for the direction, supervision and conduct of all substantive work of the IEE, including full involvement in the execution of the evaluation work. Core team members will work for extended periods from January/February 2006 to September 2007. The core team leader will be required to do some preliminary work in December 2005 and he/she and possibly some other members of the core team will be required to make inputs up to the end of November 2007.

76. Each member of the core team, including the team leader, will have responsibility for the leadership of one of the four main areas to be covered by the evaluation:

FAO’s technical work (normative, technical advisory and capacity building including in development, emergencies and rehabilitation);
management and organizational issues;
governance issues; and
FAO’s role in the multilateral system.

77. Qualifications and experience of the core team: Internationally recognized and:

at least five years’ experience at senior policy level in their field, preferably with a part of that work in developing countries;
evaluation experience, preferably including experience of complex evaluations;
significant exposure to the multilateral system;
demonstrated ability in:

leadership;
communication (written and oral);
conceptual and empirical analysis; and
synthesis reporting, including synthesis of findings and recommendations;

experience in the public sector, with experience in the private and NGO sectors being an advantage; and
at least one member of the core team will require a knowledge of quantitative and qualitative methods of social and economic research, including participatory survey techniques and cost-benefit analysis as applied to complex situations (including substantial non-quantifiable variables).

78. Evaluation core team leader: He/she will provide overall leadership of the evaluation team and have a coordinating role, as well as taking responsibility for one of the areas of specialist evaluation work. Qualifications, in addition to those above, will include:

experience of complex evaluation, preferably in the multilateral system;
knowledge of the substantive areas of FAO’s mandate;
substantial experience in a range of developing countries;
experience of strategic corporate level planning; and
knowledge of the UN and wider multilateral system.

79. Core team member (technical work of FAO): Qualifications, in addition to those above, will include an in-depth knowledge at senior level of food security issues, agriculture, rural development and a working knowledge of fisheries and forestry. This will include substantial experience in a range of developing countries.

80. Core team member (management and organizational issues): Qualifications, in addition to those above, will include a knowledge of best business management practice in a multinational and multicultural public sector context (knowledge of cutting edge private sector practice will also be a considerable advantage, as will knowledge of the UN common system). Experience will include:

programming, budgeting and results-based management;
human resource management;
financial and administrative management, including risk management and accounting standards; and
application of information and communication technology (IT/CT) in all aspects of business practice to a multi-locational organization.

81. Core team member (governance issues): Qualifications, in addition to those above, will include expertise and experience in UN and other international public sector institutions, governance issues and in institutional analysis. Experience of being a member of the governing bodies of a major international organization will be an advantage, as will experience of the involvement of non-state actors in governance.

82. Core team member (FAO’s role in the multilateral system): Qualifications, in addition to those above, will include expertise and experience of the multilateral system, particularly in FAO’s areas of mandate. Knowledge of other UN organizations, the multilateral financing institutions, the international agricultural research system and the multilateral non governmental sector will all be an advantage.

Evaluation operations administrator

83. The evaluation operations administrator will be responsible for carrying out the day-to-day administration of the IEE. The selected candidate will be required for a continuous period of service from January/February 2006 to September/October 2007. Reporting directly to the evaluation core team leader, support functions will include:

advertisement for and recruitment of consultants and other staff supporting the IEE;
contracting, travel and payments;
work-plan and budget monitoring for the core team;
managing the independent external evaluation website; and
supervising a limited number of support staff.

84. Qualifications and experience will include:

provision of operational and management support to large multilateral projects;
understanding of evaluation;
it would be an advantage to have a knowledge of:

FAO administrative and financial systems; and
FAO institutional structure, including the decentralized offices.

85. In line with FAO procedures, full payment of honoraria to all consultants engaged in the evaluation, including the core team and its leader, would be dependent on timely delivery of the outputs of the evaluation in line with the work-plan.

Annex II Appendix 2: Indicative Listing of Issues to be Addressed in the Independent External Evaluation of FAO

Background

86. The evaluation core team will define its proposals for issues to be addressed by the evaluation in its inception report. Also, during the course of the evaluation other issues may be identified which require in-depth study. It will not be possible to cover all issues in the same degree of depth and some issues will require greater analysis than others. The core team will thus make its proposals for how it intends to address the range of issues identified also as part of its inception report. To assist in this process, a preliminary indication of issues of concern to member countries for coverage in the evaluation and discussion in the finding and recommendations has been identified by the ISWG and is summarized below. In presenting the inception report, the core team is encouraged to both suggest alternative issues and exclude from consideration those issues which it regards as inappropriate. The issues are further elaborated for:

the technical work of FAO;
FAO management and organization;
FAO governance; and
FAO’s role in the multilateral system.

87. The technical work of FAO and its relevance, effectiveness and impact are at the core of the Independent External Evaluation and it will address the issues common to all evaluations, as discussed above. Issues for focus may include:

needs and priorities of members, including the extent of need of different categories of members (regions, levels of income, numbers of poor and food insecure, etc.);
technical work in areas of FAO’s mandate (food security, nutrition, overcoming rural poverty, crops, livestock, forestry, fisheries, environment and sustainable use of the natural resource base, agricultural trade, etc.);
types of work (policy, regulatory, capacity building and institutional development, scientific technical, advocacy, etc.);
level of intervention (global, regional, national, sub-national);
target group (e.g. rural poor and women);
manner of intervention (e.g. piloting, Regular Programme information outputs, advisory support);
extent to which needs met by other organizations (see FAO’s role in the multilateral system);
extent to which FAO resources and outputs are aligned with the needs and priorities of members;
process and efficiency issues for FAO’s technical work including:

integration of cross-cutting themes such as gender, sustainable livelihoods and HIV/AIDS;
integrated work across technical boundaries;
synergies between different types of work and the integration of normative and operational development work; and
partnerships with other organizations at country, regional and global levels, including partnership to build global knowledge;

quality, quantity and appropriateness of outputs, including information outputs and their dissemination;
results and sustainable impacts including the use made and implications for development at national, regional and global levels of the results of FAO work in all areas. This may give particular attention to the potential for identification of FAO specific contributions and the verification of plausible contributions to impacts in terms of human welfare from work, including but not restricted to:

the forum function for exchange of information and movement towards regional and global policy and institutional coherence in the areas of FAO’s mandate;
international agreements, treaties, regulations and standards;
policy work and policy advisory products;
advocacy;
knowledge management and information;
global and regional assessment analysis of trends (state of the sector), statistics and projections, including for trade;
piloting and demonstration;
capacity and institutional development;
resource mobilization and investment
early warning and surveillance; and
emergency response.

88. Management and organization of FAO – Issues with respect to efficiency, effectiveness, transparency, and capacity to adjust flexibly to changing needs and technological opportunities, including:

overall issues of organizational culture and practice with respect to:

transparency;
information flows, communication and reporting;
decentralization and delegation of authority;
results orientation and results-based management, including the extent to which FAO’s intended outcomes and impacts are verifiable; and
dealing with risk and uncertainty;

oversight, audit and evaluation:

coverage, adequacy and quality of analysis and reporting;
institutional arrangements including authorities, reporting lines and independence in providing information for accountability and decision-making to management and the Governing Bodies; and
arrangements in place for verifiable design and monitoring of programmes;

programming and budgeting:

processes for identifying:

strategic vision;
members’ needs and priorities; and
strengths and weaknesses including efficiency and effectiveness of programmes and comparative advantage;

proposals for resource allocation:

process (efficiency and effectiveness);
link to analysis of needs and performance; and
coherence, transparency and comprehensiveness of documentation for decision-making;

process for in-course adjustments in resource allocation and in implementation;
integration of extra-budgetary resources into the programme of work and their implications for the agreed regular programme of work financed from the core budget;
duration of the budget cycle and its implications for long-term work and efficiency; and
implications of currency movements and US dollar and Euro based budgeting.

organizational structure – appropriateness, efficiency and effectiveness to meet the needs of member countries, including:

role, structure and distribution of tasks and resources between headquarters and decentralized offices;
internal organization of headquarters, regional and other decentralized offices;
lines of reporting and authority and distribution of levels of decision-making;
work across institutional boundaries and matrix programming and management;
adequacy of communications infrastructure; and
opportunities for off-shoring and contracting out.

administrative and financial systems and procedures – including conformity to accepted international standards and issues for different sizes and categories of transaction (e.g. for emergencies and development projects) of:

purchasing and contracting including implications of procedures and authorities;
financial arrangements and procedures including arrangements for:

budget control; and
reserves and contingencies;

place of ex-post and ex-ante monitoring and control;
gaps and/or redundancies in procedures;
information and communication technology supporting systems;
opportunities at country level for:

common UN system operations; and
national execution;

resource mobilization and project systems and procedures, including:

integration with the Organization’s priorities and programming and budgeting; and
flexibility, efficiency and responsiveness;

human resource policies and procedures and their capacity to efficiently provide FAO at a competitive price with the human resources it needs of the required competencies, motivation and flexibility to meet the changing challenges of the programme, including arrangements for:

selection at all levels;
appropriate contractual arrangements;
flexible adjustment of the staff competency profile in response to changing demands;
use of short-term consultants and part-time staff and the positive and negative impacts on efficiency, flexible and appropriate delivery, recruitment of full-time staff, and on maintenance and dissemination of knowledge;
incentive structures for staff;
staff performance assessment;
staff training;
separation and retirement of staff;
ensuring geographical and gender balance; and
ensuring transparency and confidence in decision making with respect to staff.

89. FAO - Governance - Issues with respect to:

multilateral and democratic efficiency, effectiveness and inclusiveness of governance in FAO as an integral part of overall UN system governance, including:

capacity of the governance mechanisms to arrive at coherent positions and make informed decisions on major issues (including: priorities, resource levels and institutional arrangements) and factors strengthening or limiting such capacities;
capacity of the governance mechanisms for independent initiative;
extent to which Governing Body decision-making is able to take adequate account of:

national and regional needs and positions, including the role of regional bodies;
needs and positions of different sectors within FAO’s areas of mandate, including the role of the specialized committees of the Council; and
development and priorities elsewhere in the multilateral system, in particular the UN system and the General Assembly;

inclusiveness and balance in governance mechanisms and the extent to which these contribute to confidence and ownership by member country governments and the wider public, including:

capacity for all categories of member country to input to decision-making;
issues of balance in governance decision-making mechanisms within a multilateral context with respect to such issues as size of country population and size of contributions to FAO;
role of the non-governmental sector, including civil society and the private sector in governance;
role of other intergovernmental organizations, in particular of the UN;

factors for efficiency and coherence including such aspects as:

the size, frequency and duration of meetings;
gaps and/or potentials for overlap in the role of the different Governing Bodies; and
clarity of mandates of the different Governing Bodies;

factors for transparency and member confidence, including:

language policy (including efficiency implications);
availability of information; and
secretariat arrangements for the Governing Bodies (independence, competence, etc.);

information to the Governing Bodies to facilitate decision-making and its:

comprehensiveness;
clarity on issues, format and length; and
cost;

effectiveness, efficiency and coherence with respect to key areas including:

overall priority setting;
establishing resources and their application including:

the regular (core) budget of FAO (including the budget cycle and the documents presented to the Governing Bodies);
extra-budgetary funding (trust funding including the growing extra-budgetary funding for emergency work);
other potential supplementary funding arrangements (e.g. voluntary core and pool funding, multilateral trust funding);
the balance between, and complementarity of, core-budget activities and those funded by voluntary contributions;

appointment of the Director-General and officers of the Council;
institutional structure and arrangements;
approach to risk, including financial risk; and
human resource policies;

Clarity and balance in roles for decision-making between the Governing Bodies and the Director-General (actual and as set out in the Basic Texts); and
efficiency and effectiveness of the existing Governing Body structures in combining functions of governance and functions as an international forum for exchange of information and obtaining international policy and regulatory coherence in FAO’s areas of mandate.

90. FAO’s role in the multilateral system – The organizational strategy, arrangements, culture and procedures in FAO for multilateral partnership (working jointly towards synergies, reducing competition and avoiding overlaps and duplication) and awareness of the costs and benefits of multilateral partnership. In addition to bench- marking FAO performance against other comparable organizations, the evaluation may address issues with respect to FAO’s comparative advantage and role in the multilateral system, including:

the adequacy of the international architecture for FAO’s area of mandate and those areas of member country priority and need which are:

inadequately addressed at present by the international system and may require to be strengthened;
have adequate coverage without any input from FAO; and
are being ineffectively addressed, in part due to competition and/or lack of partnership between organizations.

the extent to which FAO’s areas of mandate and competence are clear and respected by other partners in the multilateral system, including the sustainability of other agencies’ intervention in areas of FAO’s mandate (e.g. the World Bank has moved heavily into and then again out of several sectoral areas covered by members of the UN system over the past two decades);
relation of FAO at the Governing Body and managerial level to other organs of the UN system, decisions of the major organs of the UN system and the UN Secretary-General;
effective partnership collaboration and integration, including:

participation and contribution in the central coordinating and partnership mechanisms of the UN system, including Chief Executives Board (CEB) and the UN Development Group (UNDG);
work at country level (including UNDAF, PRS process and collaboration in the UN country team); and
partnering with the Rome-based international food and agriculture organizations (IFAD and WFP);

And also:

iv. collaboration on treaty and international regulatory work;

v.collaboration on research and its application, including with the institutions of the CGIAR;

vi. collaborative technical programmes;

vii. collaboration in building accessible global knowledge;

viii. collaboration with non-governmental multilateral organizations; and

ix. collaboration with regional organizations.

Annex II Appendix 3: Definitions of Terms used in the Terms of Reference

Benchmark	Reference point or standard against which performance or achievements can be assessed. A benchmark often refers to the performance that has been achieved in the recent past by other comparable organizations or what can be reasonably inferred to have been achieved in the circumstances.
Effectiveness	The extent to which the intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance and the volume of resources deployed.
Efficiency	A measure of how economically resources/inputs (funds, expertise, etc.) are converted to results.
Evaluability	Extent to which an activity or a programme can be evaluated in a reliable and credible fashion. Assessing evaluability calls for an early review of a proposed activity in order to ascertain whether its objectives are adequately defined and its results verifiable.
Impacts	Positive and negative, primary and secondary long-term effects produced by an intervention, directly or indirectly, intended or unintended.
Indicator	Quantitative or qualitative factor or variable that provides a simple and reliable means to verify achievement, to reflect the changes connected to an intervention, or to help assess the performance of an actor.
Outcomes	The likely or achieved short-term and medium-term effects of an intervention’s outputs.
Outputs	The products, goods and services which result from an intervention
Performance	The degree to which an intervention or a partner operates according to specific criteria/standards/guidelines or achieves results in accordance with stated goals or plans.
Relevance	The extent to which the objectives of an intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies.
Results	The output, outcome or impact of an intervention.
Stakeholders	Agencies, organisations, groups or individuals who have a direct or indirect interest in the intervention or its evaluation.
Sustainability	The continuation of benefits from an intervention after major assistance has been completed. The probability of long-term benefits. The resilience to risk of the net benefit flows over time.
Triangulation	The use of three or more, sources or types of information, or types of analysis to verify and substantiate an assessment, in order to overcome the bias that comes from single informants, single-methods, single observer or single theory studies.

Annex II Appendix 4: Abbreviations and Acronyms

Used in the terms of reference

CGIAR

Consultative Group on International Agricultural Research

FAO

Food and Agriculture Organization of the United Nations

IEE

Independent External Evaluation

IFAD

International Fund for Agricultural Development

ISWG

Inter-Sessional Working Group of the FAO Council for the IEE

MDG

Millennium Development Goal

MTP

Medium Term Plan

OECD-DAC

Organization for Economic Cooperation and Development – Development Assistance Committee

PWB

Programme of Work and Budget

ToRs

Terms of Reference

WFP

World Food Programme

UN

United Nations

UNDAF

United Nations Development Assistance Framework

ANNEX III: THE ROLE OF THE FAO SECRETARIAT IN THE INDEPENDENT EXTERNAL EVALUATION OF FAO (IEE)

Background

91. At its Hundred and Twenty-seventh Session in November 2004, the Council instructed the ISWG to “prepare proposals for definition of an appropriate supporting role for the Secretariat” in the Independent External Evaluation (IEE).

92. In line with standard best practice, it is expected that FAO’s Secretariat and its Director-General will fully respect and honour the complete independence of the evaluation process. At the same time, it is recognized that the Secretariat including in particular the Organization’s Director-General are stakeholders in the evaluation and that the evaluation will be facilitated by their positive support. With respect to the IEE, the Evaluation Service within the FAO Secretariat is in a particular position to facilitate the work of the IEE.

The Director-General

93. The Director-General will have opportunity to provide his views to the evaluation team during the evaluation process. He will also provide to the Council his response to the conclusions and recommendations made in the evaluation report.

The Secretariat

94. The FAO Secretariat will facilitate the work of the evaluation team and of the Council Committee to oversee the evaluation without seeking to influence the work or the findings and recommendations of the evaluation. The evaluation team should be considered free to contact and discuss with any member of the Secretariat they wish within the purposes explicit in their terms of reference. In carrying out this role, the Secretariat will:

meet requests for information by the evaluation team which can be accommodated without undue adjustments in the Organization’s work programme;
support all administrative tasks necessary to the evaluation as undertaken by the evaluation administrator (including recruitment, travel, logistics, computer facilities and translation of documents);
provide office and meeting space to the evaluation team;
facilitate country visits through its decentralized offices at regional, sub-regional and country levels; and
provide support to the Council Committee for its meetings (including, translation of documents and interpretation against reimbursement).

FAO Evaluation Service

95. The Evaluation Service will play a particular facilitating role between the evaluation team and the Secretariat. It will also provide the secretariat to the Committee of the Council. It’s functions will include:

provision of the secretariat support to the Council Committee (as it has done for the ISWG);
acting as budget holder on behalf of FAO for the evaluation trust fund and provision of monitoring information on the budget and expenditures to the Council Committee (as it has done for the ISWG);
provision of a website for the evaluation; and
facilitating the evaluation team’s work particularly as regards making contacts, obtaining information and organizing visits.

Administrative and support cost considerations

96. The administration of the evaluation will be carried out in compliance with the Rules and Regulations of the Organization. However, strictly internal procedures relating to requirements for clearances at the level of the Director-General and his Deputy for fee rates, etc. and internal selection procedures for the staff and consultants of the IEE will be waived.

97. The final decision on the support cost level to be applied for the evaluation will be taken by the Council. The level of support costs for the IEE should be sufficient and should not exceed the amount needed to cover the estimated incremental cost to the Secretariat for its activities and services that support the IEE. Supporting evidence should be provided in the budget for this level⁹ (see Annex IV).

ANNEX IV: INDICATIVE BUDGET FOR THE INDEPENDENT EXTERNAL EVALUATION OF FAO

Indicative Budget of the IEE	Approximate Person months	Cost US$ (000)
Indicative Budget of the IEE	Approximate Person months	2006	2007	Total
Core team (4 persons) – (honorarium and travel to and from Rome)	80	634	633	1 267

Quality Assurance Advisers (2 persons) – (honorarium, per diem and travel)	7	76	76	152

Specialist Team Members - Work in Rome (honorarium, per diem and travel)	23	286	285	571

Country Visits, Visits to Other Agencies and country impact studies (honorarium for specialist team members, per diem and travel)	31	500	383	883

Stakeholder workshops and peer reviews		54	54	108

Support Staff (Evaluation Operations Administrator, Administrative Assistant, Temporary assistance and Research assistance)	82	310	310	620

Miscellaneous (including purchase of computers, photo-copier etc. and telephone charges)		40	10	50

Council Committee		150	240	390

FAO central administrative support *		123	119	242

Grand Total		2 173	2 110	4 283
* Includes: Office space, with electricity, cleaning, etc.; Provision of telephone, computer and internet and intranet connections; central call for funds, accounting and contracting for human resources and other contracts and purchasing, including travel. Excludes Evaluation Service and other Secretariat support

¹ CL 127/REP paras. 113-117.

² Bureau of the Chairman and seven members (one from each of the regional groups).

³ Australia, Canada, Finland, Italy, New Zealand, Sweden, Switzerland, UK and USA.

⁴ India, Netherlands, Norway and Saudi Arabia.

⁵ Supplement to the Director-General’s Programme of Work and Budget (Reform Proposals), C 2005/3Sup.1.

⁶ Monterrey International Conference on Financing for Development, 2002; Johannesburg World Summit on Sustainable Development, 2002.

⁷ Report of the Hundred and Twenty-seventh Session of the Council, November 2004, CL 127/REP paras 113-117.

⁸ Supplement to the Director-General’s Programme of Work and Budget (Reform Proposals), C 2005/3Sup.1.

⁹ As a reference, for extra-budgetary projects located in headquarters, support costs are normally set at 6 percent of the total budget, covering administrative transactions that are performed centrally (including the maintenance of central financial, contractual and other administrative systems) and the provision of office space, telephone and computer connections, etc.

ORGANIZATIONAL STRUCTURE FOR THE IEE

			IEE Council Committee │ IEE Council Committee Bureau Functions of Council Committee / Bureau: • Make all initial decisions for setting up the IEE, including, if necessary, agreement on the selection of quality advisers, core team and evaluation operations administrator if not done previously by the ISWG and FAO Council; • Provide budgetary oversight; • Agree with the inception report’s proposals regarding further detailing of methodology, areas of work and timetable for the evaluation; • Take decisions on behalf of the Council if any aspect of the evaluation requires a definitive intervention (e.g. budget shortfall, major delays), but not as regards the conduct of the evaluation; • Report progress to the Council at each session; • Receive periodic reports of work and the final report of the IEE in draft and final; and • Forward to the Council the IEE final report and a report of its own activities..
	FAO Evaluation Service Functions • secretarial support to Council Committee; • budget holder providing financial monitoring • facilitate core team work including contacts with FAO Secretariat.				Quality Assurance Advisers Functions: - To Council Committee: provide independent reports on: • quality of evaluation work and deliverables; • independence of the evaluation; • progress of activities and deliverables against schedule. -To core team: • provide, when requested, independent advice on any substantive aspect of the evaluation; • advise on the quality of the work.

		•
	Evaluation Operations Administrator Functions – Reporting to the core team leader; responsible for the administrative support to the evaluation, including: • interface with budget holder; • interface with FAO on support and logistics; • recruitments; • travel; • contacts.			•
•
•
•			Core Team Composition - core team leader, and team leaders for: governance; technical work of FAO; management and organization; and FAO’s role in the multilateral system (total 5 people). Functions – Responsible for all substantive matters of the evaluation, including full involvement in the conduct of the evaluation: • Decisions, within the agreed ToRs, on methodology for the evaluation; • Selection of supporting evaluators and research assistants; • Leadership of supporting evaluators and research assistants, including specialist teams; • Ensuring coherence of the evaluation; • Preparation of periodic reports, reports on each main area of work and the final report; Finalization of, and final decision on, all reports, including the findings and recommendations
•

CGIAR	Consultative Group on International Agricultural Research
FAO	Food and Agriculture Organization of the United Nations
IEE	Independent External Evaluation
IFAD	International Fund for Agricultural Development
ISWG	Inter-Sessional Working Group of the FAO Council for the IEE
MDG	Millennium Development Goal
MTP	Medium Term Plan
OECD-DAC	Organization for Economic Cooperation and Development – Development Assistance Committee
PWB	Programme of Work and Budget
ToRs	Terms of Reference
WFP	World Food Programme
UN	United Nations
UNDAF	United Nations Development Assistance Framework

Council

Hundred and Twenty-ninth Session

Rome, 16-18 November 2005

Independent External Evaluation of FAO

Report to the Council of the Inter-Sessional Working Group (ISWG) for the Independent External Evaluation of FAO (IEE)

Table of Contents

INTRODUCTION

WORK OF THE INTER-SESSIONAL WORKING GROUP (ISWG)

ISWG DELIVERABLES AS DEFINED BY THE COUNCIL

Terms of reference for a committee, to be established under Article VI of the FAO Constitution, to oversee, on behalf of the Council, the entire evaluation process:

Terms of reference of the Independent External Evaluation of FAO, including the scope; content; methodology; composition, expertise and criteria for selection of the evaluation team; cost estimates; reporting process; and timetable for completion of the evaluation:

Definition of an appropriate supporting role for the Secretariat

Initial arrangements to facilitate the rapid and careful selection of evaluators and any supporting experts required for the IEE (such as quality advisers)

Adequate and timely voluntary extra-budgetary funding for the evaluation

IN CONCLUSION

ANNEX I: GOVERNANCE OF THE IEE AND FUNCTIONS OF THE IEE COUNCIL COMMITTEE

The IEE Council Committee

The FAO Council

The Quality Assurance Advisers

ANNEX II: TERMS OF REFERENCE - INDEPENDENT EXTERNAL EVALUATION (IEE) OF FAO

I. Background

II. Objectives of the Evaluation

III. The Evaluation Team and its Role

IV. Scope of the Evaluation

Key evaluation issues

Period covered by the evaluation

V. Methodology of the Evaluation

Maximising the use of existing information

Assessing impacts

A consultative and transparent process

Samples for in-depth evaluation and analysis

Consideration of other institutions and benchmarking

Recommendations of the IEE

VII. Deliverables and Timetable

Annex II Appendix 1: Required Qualifications for the Evaluation Core Team and Evaluation Operations Administrator

Evaluation operations administrator

Annex II Appendix 2: Indicative Listing of Issues to be Addressed in the Independent External Evaluation of FAO

Background

Annex II Appendix 3: Definitions of Terms used in the Terms of Reference

Annex II Appendix 4: Abbreviations and Acronyms

Used in the terms of reference

ANNEX III: THE ROLE OF THE FAO SECRETARIAT IN THE INDEPENDENT EXTERNAL EVALUATION OF FAO (IEE)

Background

The Director-General

The Secretariat

FAO Evaluation Service

Administrative and support cost considerations

ANNEX IV: INDICATIVE BUDGET FOR THE INDEPENDENT EXTERNAL EVALUATION OF FAO