PC 87/6


Programme Committee

Eighty-seventh Session

Rome, 6-10 May 2002

A Progress Report on the Development of Indicators

Table of Contents


Introduction

The Approach to Indicators

Indicators and the Results-Based Model

Monitoring the FAO Gender Development Plan of Action 2002-07 and Other Key Areas

Recent Steps Taken by the Organization to Improve Application of Indicators in Programme Entities

Application of Indicators - A Learning Process

Conclusion

Annex I

Annex II


INTRODUCTION

1. At its Hundred and Nineteenth Session in November 2000, the Council welcomed the new format of the Medium-term Plan (MTP) and the extensive evidence of solid links between the Corporate Strategies and Strategic Objectives in the Strategic Framework 2000-2015. The Council also "endorsed the innovation of presenting a range of achievement indicators to facilitate monitoring of progress in the implementation of the programme entities contained in the Plan. However, while recognising that this was not always straightforward, it considered that there was room to identify more precise and measurable indicators, particularly in the case of technical projects".

2. The present paper considers the steps taken by the Organization to address this difficult area, which is being treated as an integral part of the results-based approach to programme planning, monitoring and evaluation. The improvement of indicators of outcomes and achievement of objectives for impact is being given particular attention. The ultimate concern is to verify the contribution of FAO's work to the Objectives of Member Nations as defined in the Strategic Framework.

3. This document addresses the application of indicators in the technical and economic programmes of the Organization in the context of results-based programming and the new programming model. The model itself was most recently summarised in the MTP 2002-07 paras. 60-65 and definitions and explanations are provided in Annex II to this paper. The assessment of the progress on cross-organizational strategies as defined in the Strategic Framework will also receive attention by the Organization. However, this is of less immediate concern in the context of programme planning and management improvements (which in themselves directly contribute to several of the cross-organizational strategies, including: "ensuring excellence"; "leveraging resources" and "communicating FAO's messages").

THE APPROACH TO INDICATORS

4. Indicators are intended to provide tangible evidence of achievement. They can be both quantitative and qualitative and are generally a variable which can be objectively verified (measured or assessed) to determine the extent to which the outcomes and objectives are achieved. It follows that they should help demonstrate the causality between FAO's action, the Outcome and the Objective.

5. For an indicator to be meaningful, it is necessary to specify (in addition to the indicator itself) how verification will be carried out, the "Means of Verification". For clarity of definition, FAO maintains distinction between the statements of: a) the indicators used to assess achievement; b) the planned achievement itself (including the targets specified in Objective and Outcome statements); and c) the means of assessing the indicator (means of verification).

6. Indicators are required on: the extent to which resources and the outputs delivered contribute to the priorities as defined by Member Nations; the eventual impacts of those outputs; and the extent to which targets were met. Indicators are approached at several levels:

INDICATORS AND THE RESULTS-BASED MODEL

7. Programme entities are the main building blocks of the Organization's technical and economic programmes and each should have a clear objective and point in a common direction, contributing to the Strategic Framework Strategies. Programme entities are defined in the rolling six-year Medium-term Plan which is developed each biennium. Details of the biennial programme of work to be carried out under each entity are formulated in the two-year Programme of Work and Budget. There are three main types of programme entity: technical projects (TPs); continuing programme activities (CPs) and technical service agreements (TSs):

Application of Indicators to Technical Projects (TP) and Continuing Programme Activities (CP)

Table 1: Summary of the Application of Indicators in TP and CP Programme Entities

The Hierarchy of Means-Ends as specified in TPs and CPs

The Indicators

Programme Entity Rationale: Justification in terms of the Strategic Framework and the problem to be addressed, including, how a contribution will be made to the beneficiaries (i.e. people such as producers, consumers or the public at large) who are expected to ultimately benefit.

Indicators not generally pre-defined. Assessment made through in-depth qualitative and quantitative evaluation on themes, programmes and strategies.

Programme Entity Objective: The unifying statement of the impact that the programme entity will have beyond the immediate use to which the Major Outputs are put. This will specify the secondary users of Major Output Outcomes and the expected result of them taking up the Outcomes of the Major Output from the primary users.

Indicators defined in the MTP as part of the formulation process. Principally assessed through auto-evaluation at least every six years. Also assessed in independent evaluations reported in the Programme Evaluation Report (PER)

Programme Entity Major Output Outcomes: When the Major Output (see below) is taken and used by someone else outside FAO, it has an immediate Outcome. The users are specified as well as how they will use the Major Output, with what anticipated immediate result.

Indicators defined in the MTP as part of the formulation process. Principally reviewed through the auto-evaluation and biennial assessment processes.

Programme Entity Major Outputs: The products and services delivered by FAO to specific Users. These are either an aggregation of a series of other more minor outputs to form an end product or a single major end product.

The target is essentially the product/service description including quantity, quality and to whom it will be delivered, where. The extent to which targets are met will be monitored through the biennial assessments.

Biennial Outputs: These are listed in the Programme of Work and Budget. They contribute to a single major output, and are distinct products and services produced in a biennium. Normally, they will be products for external use, but sometimes the significant internal progress of the biennium towards delivery of a major output in a subsequent biennium may be listed, as may be a significant product for internal FAO use.

The target is essentially the product/service description including scope and quantity. The extent to which targets are met are currently monitored and reported in the PIR database.

8. It has been important to clarify in the application of results-based programming, the levels at which indicators can reasonably be specified during planning and where at a later stage more complex processes of analysis and judgement, through evaluation, will be essential in arriving at a meaningful assessment of results.

9. It will be noted from the table above that there is a very long line of causality between the delivery to primary users of many FAO normative products and an eventual benefit to people or future generations in such areas as the environment (ultimate beneficiaries). This means that only a complex evaluation process can assess whether there is reasonable expectation of end impact. By illustration, the FAO Secretariat may assist Member Nations in developing a code with environmental benefits but for that code to have any result, it will need to be adapted nationally, translated into legislation and institutional arrangements and then be implemented in each country. Similarly an assessment of commodities in relation to trade will need to be utilised by members in developing their trade policies; those policies will need to be implemented; and will need to have the desired positive result in terms of benefit to their populations.

10. Indicators are specified in advance at the level of:

11. For example, if a Major Output is a comprehensive package of guidelines, training and access to electronic resource materials in a particular subject matter area, this Major Output will be delivered to a group of practitioners in government, universities and the development community. They in turn may further disseminate the Major Output and also modify it for national application in transferring it to those who will actually apply it, such as local NGOs, extension staff or trainers. This secondary level of use will be assessed through indicators applied at the planning stage but will also rely on a qualitative process of evaluation for assessment (i.e. through auto-evaluation of all programme entities and independent evaluation of selected programmes and themes), whereas the first level of use should be more directly monitorable through selected indicators.

Application of Indicators to Technical Service Agreements (TS)

12. Technical Service Agreement activities and outputs will be recorded and reported in the PIR, including pertinent aspects such as geographical and Strategic Framework focus. In addition, the contribution of TS to achievement of impacts will be specifically covered during the evaluation process for the related TPs and CPs.

Application of Indicators in Evaluating at the Level of Strategic Framework Strategies

13. The planned contribution of each programme entity to the Strategic Objectives and the subordinate Strategies is recorded in the MTP planning system. Effectiveness of work in pursuit of the Strategic Objectives will then be assessed by independent evaluation, utilising the outcome and objective indicators for the pertinent programme entities.

MONITORING THE FAO GENDER DEVELOPMENT PLAN OF ACTION 2002-07 AND OTHER KEY AREAS

Gender Development Plan of Action

14. The Thirty-first Session of the Conference in November 2001 approved the FAO Gender and Development Plan of Action 2002-07 (C 2001/9). This explicitly places the mainstreaming of gender in development in the MTP process and provides for gender to be integrated into the programme entity indicators. The system will thus track the contribution planned to be made in this area by each programme entity, which has been related to the Plan of Action, and progress recorded in the indicators. This information will be available to the Gender and Population Division through PIRES (see para. 16 below) for analysis and reporting to the Governing Bodies.

Other Areas

15. As has been the case with training, it is envisaged that the Governing Bodies will identify from time-to-time areas on which they require more information. Appropriate indicators will be identified and integrated within the programme entity based monitoring and evaluation system.

RECENT STEPS TAKEN BY THE ORGANIZATION TO IMPROVE APPLICATION OF INDICATORS IN PROGRAMME ENTITIES

Programme Planning, Implementation Reporting and Evaluation Support System (PIRES)

16. The computerised Programme Planning, Implementation Reporting and Evaluation Support System (PIRES) is a user-friendly application, intended to serve divisional and departmental management while facilitating their input to corporate plans and reporting. The first component of the system to be released was that for preparation of the Medium-term Plan. It provides an interface for the development of programme entities, including indicators. The system incorporates guidance and help and this is being further strengthened for the future with examples.

17. Within the PIRES system, managers will also be provided with working level applications to better track production of outputs and services through the Departmental Programme Management Tool Kit. At a later date, biannual assessment reporting will be facilitated, including the tracking of Major Output Outcome indicators.

Training

18. Training has been provided to 44 resource persons in the technical and economic departments to enable them to assist their colleagues in improving programme entity formulation, with particular respect to entity statements, indicators and their means of verification. A core group within PBE has also received intensive training. Development of an intra-net based distance learning course is now under consideration. This would provide an opportunity for a wider group of staff to receive training for the future and also for expansion of the help and guidance available in PIRES.

Auto-evaluation and Annual Assessment

19. In November 2001 the Director-General issued DGB 2001/33 "Strengthening the FAO Evaluation System". Guidelines for auto-evaluation and annual assessment are shortly to be issued within the framework of this DGB. These will require TPs and CPs to be evaluated by managers, at least every six years including reporting on indicators. In addition, managers will be required to carry out annual and biannual assessments.

APPLICATION OF INDICATORS - A LEARNING PROCESS

20. In developing its results-based programming approach, beginning from 1996, the Organization has been learning by carefully reviewing the experience of national governments and other international organizations. Earlier this year FAO completed an internal management review of its own experience in the light of experience with the performance reports of the US Federal Agencies.

21. Benefit has also been taken of the lessons learned from other efforts to improve planning techniques such as the donor-funded review of indicators for the FAO/Netherlands Partnership Programme and the work of external consultants on the gender-specific indicators.

22. It is very apparent that where indicators are applied outside non-line agencies which deliver specific benefits directly to people, such as health, education and extension services, there are major problems of ascertaining end-impact at the level of people (e.g. producers, consumers, the environment). This is even more the case with an international agency which delivers normative products which have to be subsequently taken-up at country level or applied at international level in such a way as to benefit countries. The cost of systematically compiling information on outcomes and impacts also becomes proportionately greater as the number of links in the chain of causality increases.

23. Another difficulty is posed by the timeframe. Although Major Output-Outcome indicators will be included in the biennial assessment process, this cannot be for all TPs each biennium as Major Outputs are built-up over more than one biennium before a substantive product starts to become available. Real progress on objectives will often ensue after a TP has been completed and secondary users are applying the outcomes. The auto-evaluation and independent evaluation processes are thus critical.

24. Insistence on verifiable indicators can also lead to a risk that should be avoided; that is of not doing the things which are most important because the results are difficult to verify. In the light of experience, guidance on the realistic identification of a limited number of verifiable indicators is being given to Departments. The Departments' own initiatives and practice in actually monitoring and evaluating those indicators will enable a gradual improvement. Analysis of representative programme entities has allowed the Organization to start developing typical examples of indicators. The principles and guidance offered include:

25. Examples of the types of indicators now being identified through best practice are provided in Annex I to this paper. Examination of these will show that almost all are, in part, qualitative. It is doubtful if it will ever be possible to utilise fully standard indicators which can be summated in any way at the Outcome and Objective levels. This will constrain routine reporting on achievements against Strategies; rather reporting will also need to take advantage of examples and the application of analytical judgement.

CONCLUSION

26. This document is for information as requested by the Committee. While no decisions are sought, any guidance that the Committee may wish to provide would be welcomed.

 

Annex I

 

Examples of the Types of Indicators Actually Developed for Programme Entities in FAO Best-Practice

Systems, methods and approaches for management of production and of natural resources by producers and communities (including pilot activities, guidelines, decision support systems and demonstration) - Work often falling in Strategies A, C and D

Objective Level

  • Listing of national or international programmes which have taken up and adapted methodologies to local use, including indications of scale in factors such as: the numbers of people likely to be affected; and food, poverty, gender, economic, or environmental benefits.

Major Output - Outcome Level

  • Level of demand for copies and number of hits on web pages from various categories of users;

  • Use as assessed by readership surveys;

  • References and dissemination in specialised media;

  • Requests for additional information;

  • Use in field projects;

  • Feedback on uptake during demonstrations.

National capacity building, including case study examples, guidelines and training to develop institutional capacity and organizational development and establishment of national programmes - Work in all Strategies

Objective Level

  • Examples of the improvements made in individual countries as a result of the capacity improvements

Major Output - Outcome Level

  • Feedback on extent to which persons trained have been able to apply their training;

  • Examples of institutional and managerial changes, etc. making use of guidelines and methods;

  • Evidence of use made of guidelines methods etc. in terms of hits on web-sites, requests for additional information;

  • Further dissemination through citations in the literature, use in training and education curricula, adoption and adaptation in their work by other international development agencies;

  • Use made in FAO projects.

Support for internationally agreed guidelines, codes, undertakings norms and standards - Work often in Strategy B

Objective Level

  • Adoption/adaptation of codes, standards, etc. in national legislation and policy;

  • Reference to codes, standards, etc., in other international instruments;

  • Reduction in disputes notified between governments on the application and use of international standards (e.g. to WTO for Trade).

Major Output - Outcome Level

  • Volume/coverage of standards, codes, etc. adopted and in approval processes by international bodies;

  • Level of demand for copies and number of hits on web pages from various categories of users.

National policy, strategy and legislative development including case study examples, identification of issues, guidelines and training - Work in Strategy B

Objective Level

  • Examples of countries' promulgation of the policy, legislation, etc., including indications of scale, such as areas involved, population who may benefit, and implications for poverty and food security.

Major Output - Outcome Level

  • Improved policies, strategies or legislation developed;

  • Other international agencies recommending the same approach based on the work;

  • Citations in literature, discussion in international and regional meetings, etc.;

  • Use in training and education.

Support to international and regional collaborative arrangements - Work in all Strategies but often in Strategy D for Shared Natural Resource Management

Objective Level

  • Examples of concrete results from decisions or work of the collaborative fora, e.g. for management of natural resources such as fish, including indications of scale, such as areas involved and population who may benefit.

Major Output - Outcome Level

  • Numbers and nature of participants in the fora;

  • Users' feedback on the usefulness of meetings and fora;

  • References to the work and outcomes of fora in specialist and general media;

  • Examples of major lessons learned/agreed in coordination fora;

  • Collaborative work ensuing from discussions in fora.

Global and regional information and monitoring systems and outlook studies and analyses including the State of the World and AT Publications - Work often falling in Strategy E

Objective Level

  • International and national initiatives which make explicit use of the information;

  • Survey and evaluation of importance attached by national and international organizations to this information and the use they make of it.

Major Output - Outcome Level

  • Level of demand for copies and number of hits on web pages from various categories of users;

  • Use as assessed by a readership surveys;

  • References in specialised and general media;

  • Requests for related information and additional analysis by national authorities and secretariats and governing bodies of international organizations and secretariats of international agreements.

 

Annex II

DEFINITIONS AND EXPLANATION OF TERMS FOR MTP PROGRAMME ENTITIES AND INDICATORS

Programme Entity

27. Programme entities are the main building blocks of the Organization's technical and economic programmes and should each have a clear objective and point in a common direction, contributing to the Strategic Framework strategies. Programme entities are defined in the rolling six-year Medium-term Plan which is developed each biennium. Details of the work to be done under each entity are formulated in the two-year Programme of Work and Budget. For the Technical and Economic Programmes of the Organization, there are three main types of programme entity: technical projects (TPs); continuing programme activities (CPs) and technical service agreements (TSs):

    1. Technical Project (TP): A TP is to produce a defined set of major outputs to make a clearly verifiable contribution to an objective at the level of FAO member countries or the international community. It has a maximum time horizon of six years (but preferably less) and requires an implementation plan covering the whole of the TP duration.
    2. Continuing Programme Activity (CP): A CP is similar to a TP, but covers major outputs of a recurrent nature, such as statistical series, recurrent situation and outlook studies and servicing of conventions, agreements, etc. CPs are not intended for production of discrete outputs in a subject matter area believed to be of continuing importance. For this, the contribution which will be made in a fixed timeframe should be defined in a TP.
    3. Technical Service Agreement (TS): A TS allows for planned allocation of resources for work which is demand-driven and where the individual outputs are likely to be discrete tasks. There are three major types of TS:

Programme Entity Rationale

28. This is a statement of the purpose of the programme entity (beyond the Objective statement - see below). It will address four areas:

    1. justification for the programme entities in terms of the Strategic Framework and its strategic objectives;
    2. the problem to be addressed through the programme entity and how benefits will result for Beneficiaries;
    3. why it is reasonable to believe that achievement of the programme entity Objective will deliver a benefit to the identified beneficiaries. What is the line of causality (means-ends) between the Objective of the programme entity and the problem that the entity is intended to ultimately address; and
    4. why is there a priority for FAO in carrying out this work and in what way does the Organization have a comparative advantage in this work.

Beneficiaries

29. Beneficiaries are the people (producers, consumers, the public at large) who are expected to ultimately benefit from the development process in which the programme entity is making a contribution. This can include the international community. If any Vulnerable Groups are particularly intended to benefit, they should be referred to at this level. Vulnerable Groups have been classified using the original FIVIMS categories, and include members of low-income households within vulnerable livelihood systems. Beneficiaries are also classified by Gender (male, female or both).

Programme Entity Objective

30. The Objective is the unifying statement of the impact or result that the programme entity will have beyond the immediate use to which the Major Outputs (see below) are put, and is the link to the Rationale. To the maximum extent possible, major outputs should contribute towards a single objective, but there may be occasions where the objective is justifiably broken down into two or three impacts, where the Outcomes of the major outputs are contributing to addressing the Problem in the Rationale in more than one way. The programme entity Objective should be stated in terms of its expected benefits. It is not something FAO itself produces and cannot be the same thing as the Outcomes of the major output (see below) or the Rationale above. In terms of sequence, the result or change expected at the objective level should be achieved in member country institutions, the international community or development partners, building on the outcomes of the major outputs. It represents one significant step forward in the line of causality between the major output Outcomes and the Rationale, and, in this sense, can be conceptualised as a "second-degree" effect of the major outputs.

31. The Objective should specify:

    1. who will be acting to deliver benefits based on or using the Outcomes of the major outputs. As an Objective will seldom refer to benefits at the level of Beneficiaries, i.e. rural people, consumers, etc. (these are in the Rationale), it will normally define who in the chain of causality is taking the benefit to the next step (e.g. extension workers, academics, engineers or policy makers);
    2. the nature of the benefit and its scope/size - targets (in terms of parts of the world, numbers of institutions, etc.); and
    3. when this benefit is expected to be realised, e.g. shortly (normally within a couple of years of the programme entity being completed).

Programme Entity Major Outputs

32. Programme Entity Major Outputs are overall products and services delivered by FAO to specific Users, and are either an aggregation of a series of other more minor outputs to form an end-product or a single major end-product. SOFA or a fully-developed and tested methodology for natural resource assessment are examples of major end-products. A set of curricula, training materials, guidelines and methods for priority aspects of coastal fisheries management piloted and introduced in Small Island States is an aggregation of outputs. A major output is not open-ended, and is completed by the end of a TP or probably each biennium in a CP. In a TP, the major output can be the same as a "biennial output" (see below) in the year in which it is finalised, but this is unlikely. In a CP, the major output and the biennial output are more likely to be the same thing (e.g. SOFA).

Major Output Outcomes

33. The major output is a product, service or combination of the two. It is produced by FAO. When it is taken and used by someone else outside FAO (i.e. its "Users"), it has an immediate Outcome. The Outcome will specify what the major output is to be used for, with what expected immediate result. The scope in terms of such factors as numbers of Users who will be applying it to reach what size of audience (targets) should also be specified.

Users

34. Users are the direct recipients of the major output, who will put it to work to achieve the outcome. They are people (engineers, policy economists, etc.) in institutions (government departments, donors, universities, etc.). The major output must be able to reach directly its intended Users. If a gap exists in reaching the intended Users, it must be addressed in the major output.

Geographic Focus

35. The geographic focus refers to where the major output will contribute to the programme entity objectives. This will normally, but not always, be the same place as it has its Outcome and where the Users are located.

Country Socio-Economic Classification

36. In addition to geographic focus, major outputs can be coded by types of country within the category "All Developing Regions", e.g.: least developed; small island; and low-income, food deficit.

Biennial Outputs

37. Biennial Outputs are listed in the Programme of Work and Budget. They contribute to a single major output, and are distinct products and services or groups of similar products and services, produced in a biennium. Normally, they will be products for external use, but sometimes the significant internal progress of the biennium towards delivery of a major output in a subsequent biennium may be listed, as may be a significant product for internal FAO use.

Indicators

38. Indicators are requested on the Outcomes of major outputs and the programme entity Objective. An indicator provides tangible evidence of achievement of the outcome or objective and is generally a variable which can be objectively verified (measured or assessed) to determine the extent to which the Outcome or Objective is achieved, and must refer directly to it, i.e. demonstrating the causality between FAO's action and the Outcome or Objective. The Indicator is not, however, a restatement of the Outcome or Objective, and it is not a target. Indicators can be both quantitative and qualitative.

Means of Verification

39. For an Indicator to be meaningful, it is necessary to specify (in addition to the Indicator itself) how verification will be done. This must be practicable, but may require some resources (e.g. for survey or assessing references made to information).