PC
87/6  |
| PC
87/6 |
Eighty-seventh Session |
Rome, 6-10 May 2002 |
A Progress Report on the Development of Indicators |
Indicators and the Results-Based Model
Monitoring the FAO Gender Development Plan of Action 2002-07 and Other Key Areas
Recent Steps Taken by the Organization to Improve Application of Indicators in Programme Entities
1. At its Hundred and Nineteenth Session in November 2000, the Council welcomed the new format of the Medium-term Plan (MTP) and the extensive evidence of solid links between the Corporate Strategies and Strategic Objectives in the Strategic Framework 2000-2015. The Council also "endorsed the innovation of presenting a range of achievement indicators to facilitate monitoring of progress in the implementation of the programme entities contained in the Plan. However, while recognising that this was not always straightforward, it considered that there was room to identify more precise and measurable indicators, particularly in the case of technical projects".
2. The present paper considers the steps taken by the Organization to address this difficult area, which is being treated as an integral part of the results-based approach to programme planning, monitoring and evaluation. The improvement of indicators of outcomes and achievement of objectives for impact is being given particular attention. The ultimate concern is to verify the contribution of FAO's work to the Objectives of Member Nations as defined in the Strategic Framework.
3. This document addresses the application of indicators in the technical and economic programmes of the Organization in the context of results-based programming and the new programming model. The model itself was most recently summarised in the MTP 2002-07 paras. 60-65 and definitions and explanations are provided in Annex II to this paper. The assessment of the progress on cross-organizational strategies as defined in the Strategic Framework will also receive attention by the Organization. However, this is of less immediate concern in the context of programme planning and management improvements (which in themselves directly contribute to several of the cross-organizational strategies, including: "ensuring excellence"; "leveraging resources" and "communicating FAO's messages").
4. Indicators are intended to provide tangible evidence of achievement. They can be both quantitative and qualitative and are generally a variable which can be objectively verified (measured or assessed) to determine the extent to which the outcomes and objectives are achieved. It follows that they should help demonstrate the causality between FAO's action, the Outcome and the Objective.
5. For an indicator to be meaningful, it is necessary to specify (in addition to the indicator itself) how verification will be carried out, the "Means of Verification". For clarity of definition, FAO maintains distinction between the statements of: a) the indicators used to assess achievement; b) the planned achievement itself (including the targets specified in Objective and Outcome statements); and c) the means of assessing the indicator (means of verification).
6. Indicators are required on: the extent to which resources and the outputs delivered contribute to the priorities as defined by Member Nations; the eventual impacts of those outputs; and the extent to which targets were met. Indicators are approached at several levels:
7. Programme entities are the main building blocks of the Organization's technical and economic programmes and each should have a clear objective and point in a common direction, contributing to the Strategic Framework Strategies. Programme entities are defined in the rolling six-year Medium-term Plan which is developed each biennium. Details of the biennial programme of work to be carried out under each entity are formulated in the two-year Programme of Work and Budget. There are three main types of programme entity: technical projects (TPs); continuing programme activities (CPs) and technical service agreements (TSs):
Table 1: Summary of the Application of Indicators in TP and CP Programme Entities | |
The Hierarchy of Means-Ends as specified in TPs and CPs |
The Indicators |
Programme Entity Rationale: Justification in terms of the Strategic Framework and the problem to be addressed, including, how a contribution will be made to the beneficiaries (i.e. people such as producers, consumers or the public at large) who are expected to ultimately benefit. |
Indicators not generally pre-defined. Assessment made through in-depth qualitative and quantitative evaluation on themes, programmes and strategies. |
Programme Entity Objective: The unifying statement of the impact that the programme entity will have beyond the immediate use to which the Major Outputs are put. This will specify the secondary users of Major Output Outcomes and the expected result of them taking up the Outcomes of the Major Output from the primary users. |
Indicators defined in the MTP as part of the formulation process. Principally assessed through auto-evaluation at least every six years. Also assessed in independent evaluations reported in the Programme Evaluation Report (PER) |
Programme Entity Major Output Outcomes: When the Major Output (see below) is taken and used by someone else outside FAO, it has an immediate Outcome. The users are specified as well as how they will use the Major Output, with what anticipated immediate result. |
Indicators defined in the MTP as part of the formulation process. Principally reviewed through the auto-evaluation and biennial assessment processes. |
Programme Entity Major Outputs: The products and services delivered by FAO to specific Users. These are either an aggregation of a series of other more minor outputs to form an end product or a single major end product. |
The target is essentially the product/service description including quantity, quality and to whom it will be delivered, where. The extent to which targets are met will be monitored through the biennial assessments. |
Biennial Outputs: These are listed in the Programme of Work and Budget. They contribute to a single major output, and are distinct products and services produced in a biennium. Normally, they will be products for external use, but sometimes the significant internal progress of the biennium towards delivery of a major output in a subsequent biennium may be listed, as may be a significant product for internal FAO use. |
The target is essentially the product/service description including scope and quantity. The extent to which targets are met are currently monitored and reported in the PIR database. |
8. It has been important to clarify in the application of results-based programming, the levels at which indicators can reasonably be specified during planning and where at a later stage more complex processes of analysis and judgement, through evaluation, will be essential in arriving at a meaningful assessment of results.
9. It will be noted from the table above that there is a very long line of causality between the delivery to primary users of many FAO normative products and an eventual benefit to people or future generations in such areas as the environment (ultimate beneficiaries). This means that only a complex evaluation process can assess whether there is reasonable expectation of end impact. By illustration, the FAO Secretariat may assist Member Nations in developing a code with environmental benefits but for that code to have any result, it will need to be adapted nationally, translated into legislation and institutional arrangements and then be implemented in each country. Similarly an assessment of commodities in relation to trade will need to be utilised by members in developing their trade policies; those policies will need to be implemented; and will need to have the desired positive result in terms of benefit to their populations.
10. Indicators are specified in advance at the level of:
11. For example, if a Major Output is a comprehensive package of guidelines, training and access to electronic resource materials in a particular subject matter area, this Major Output will be delivered to a group of practitioners in government, universities and the development community. They in turn may further disseminate the Major Output and also modify it for national application in transferring it to those who will actually apply it, such as local NGOs, extension staff or trainers. This secondary level of use will be assessed through indicators applied at the planning stage but will also rely on a qualitative process of evaluation for assessment (i.e. through auto-evaluation of all programme entities and independent evaluation of selected programmes and themes), whereas the first level of use should be more directly monitorable through selected indicators.
12. Technical Service Agreement activities and outputs will be recorded and reported in the PIR, including pertinent aspects such as geographical and Strategic Framework focus. In addition, the contribution of TS to achievement of impacts will be specifically covered during the evaluation process for the related TPs and CPs.
13. The planned contribution of each programme entity to the Strategic Objectives and the subordinate Strategies is recorded in the MTP planning system. Effectiveness of work in pursuit of the Strategic Objectives will then be assessed by independent evaluation, utilising the outcome and objective indicators for the pertinent programme entities.
14. The Thirty-first Session of the Conference in November 2001 approved the FAO Gender and Development Plan of Action 2002-07 (C 2001/9). This explicitly places the mainstreaming of gender in development in the MTP process and provides for gender to be integrated into the programme entity indicators. The system will thus track the contribution planned to be made in this area by each programme entity, which has been related to the Plan of Action, and progress recorded in the indicators. This information will be available to the Gender and Population Division through PIRES (see para. 16 below) for analysis and reporting to the Governing Bodies.
15. As has been the case with training, it is envisaged that the Governing Bodies will identify from time-to-time areas on which they require more information. Appropriate indicators will be identified and integrated within the programme entity based monitoring and evaluation system.
16. The computerised Programme Planning, Implementation Reporting and Evaluation Support System (PIRES) is a user-friendly application, intended to serve divisional and departmental management while facilitating their input to corporate plans and reporting. The first component of the system to be released was that for preparation of the Medium-term Plan. It provides an interface for the development of programme entities, including indicators. The system incorporates guidance and help and this is being further strengthened for the future with examples.
17. Within the PIRES system, managers will also be provided with working level applications to better track production of outputs and services through the Departmental Programme Management Tool Kit. At a later date, biannual assessment reporting will be facilitated, including the tracking of Major Output Outcome indicators.
18. Training has been provided to 44 resource persons in the technical and economic departments to enable them to assist their colleagues in improving programme entity formulation, with particular respect to entity statements, indicators and their means of verification. A core group within PBE has also received intensive training. Development of an intra-net based distance learning course is now under consideration. This would provide an opportunity for a wider group of staff to receive training for the future and also for expansion of the help and guidance available in PIRES.
19. In November 2001 the Director-General issued DGB 2001/33 "Strengthening the FAO Evaluation System". Guidelines for auto-evaluation and annual assessment are shortly to be issued within the framework of this DGB. These will require TPs and CPs to be evaluated by managers, at least every six years including reporting on indicators. In addition, managers will be required to carry out annual and biannual assessments.
20. In developing its results-based programming approach, beginning from 1996, the Organization has been learning by carefully reviewing the experience of national governments and other international organizations. Earlier this year FAO completed an internal management review of its own experience in the light of experience with the performance reports of the US Federal Agencies.
21. Benefit has also been taken of the lessons learned from other efforts to improve planning techniques such as the donor-funded review of indicators for the FAO/Netherlands Partnership Programme and the work of external consultants on the gender-specific indicators.
22. It is very apparent that where indicators are applied outside non-line agencies which deliver specific benefits directly to people, such as health, education and extension services, there are major problems of ascertaining end-impact at the level of people (e.g. producers, consumers, the environment). This is even more the case with an international agency which delivers normative products which have to be subsequently taken-up at country level or applied at international level in such a way as to benefit countries. The cost of systematically compiling information on outcomes and impacts also becomes proportionately greater as the number of links in the chain of causality increases.
23. Another difficulty is posed by the timeframe. Although Major Output-Outcome indicators will be included in the biennial assessment process, this cannot be for all TPs each biennium as Major Outputs are built-up over more than one biennium before a substantive product starts to become available. Real progress on objectives will often ensue after a TP has been completed and secondary users are applying the outcomes. The auto-evaluation and independent evaluation processes are thus critical.
24. Insistence on verifiable indicators can also lead to a risk that should be avoided; that is of not doing the things which are most important because the results are difficult to verify. In the light of experience, guidance on the realistic identification of a limited number of verifiable indicators is being given to Departments. The Departments' own initiatives and practice in actually monitoring and evaluating those indicators will enable a gradual improvement. Analysis of representative programme entities has allowed the Organization to start developing typical examples of indicators. The principles and guidance offered include:
25. Examples of the types of indicators now being identified through best practice are provided in Annex I to this paper. Examination of these will show that almost all are, in part, qualitative. It is doubtful if it will ever be possible to utilise fully standard indicators which can be summated in any way at the Outcome and Objective levels. This will constrain routine reporting on achievements against Strategies; rather reporting will also need to take advantage of examples and the application of analytical judgement.
26. This document is for information as requested by the Committee. While no decisions are sought, any guidance that the Committee may wish to provide would be welcomed.
Examples of the Types of Indicators Actually Developed for Programme Entities in FAO Best-Practice | |
Systems, methods and approaches for management of production and of natural resources by producers and communities (including pilot activities, guidelines, decision support systems and demonstration) - Work often falling in Strategies A, C and D | |
Objective Level |
|
Major Output - Outcome Level |
|
National capacity building, including case study examples, guidelines and training to develop institutional capacity and organizational development and establishment of national programmes - Work in all Strategies | |
Objective Level |
|
Major Output - Outcome Level |
|
Support for internationally agreed guidelines, codes, undertakings norms and standards - Work often in Strategy B | |
Objective Level |
|
Major Output - Outcome Level |
|
National policy, strategy and legislative development including case study examples, identification of issues, guidelines and training - Work in Strategy B | |
Objective Level |
|
Major Output - Outcome Level |
|
Support to international and regional collaborative arrangements - Work in all Strategies but often in Strategy D for Shared Natural Resource Management | |
Objective Level |
|
Major Output - Outcome Level |
|
Global and regional information and monitoring systems and outlook studies and analyses including the State of the World and AT Publications - Work often falling in Strategy E | |
Objective Level |
|
Major Output - Outcome Level |
|
27. Programme entities are the main building blocks of the Organization's technical and economic programmes and should each have a clear objective and point in a common direction, contributing to the Strategic Framework strategies. Programme entities are defined in the rolling six-year Medium-term Plan which is developed each biennium. Details of the work to be done under each entity are formulated in the two-year Programme of Work and Budget. For the Technical and Economic Programmes of the Organization, there are three main types of programme entity: technical projects (TPs); continuing programme activities (CPs) and technical service agreements (TSs):
28. This is a statement of the purpose of the programme entity (beyond the Objective statement - see below). It will address four areas:
29. Beneficiaries are the people (producers, consumers, the public at large) who are expected to ultimately benefit from the development process in which the programme entity is making a contribution. This can include the international community. If any Vulnerable Groups are particularly intended to benefit, they should be referred to at this level. Vulnerable Groups have been classified using the original FIVIMS categories, and include members of low-income households within vulnerable livelihood systems. Beneficiaries are also classified by Gender (male, female or both).
30. The Objective is the unifying statement of the impact or result that the programme entity will have beyond the immediate use to which the Major Outputs (see below) are put, and is the link to the Rationale. To the maximum extent possible, major outputs should contribute towards a single objective, but there may be occasions where the objective is justifiably broken down into two or three impacts, where the Outcomes of the major outputs are contributing to addressing the Problem in the Rationale in more than one way. The programme entity Objective should be stated in terms of its expected benefits. It is not something FAO itself produces and cannot be the same thing as the Outcomes of the major output (see below) or the Rationale above. In terms of sequence, the result or change expected at the objective level should be achieved in member country institutions, the international community or development partners, building on the outcomes of the major outputs. It represents one significant step forward in the line of causality between the major output Outcomes and the Rationale, and, in this sense, can be conceptualised as a "second-degree" effect of the major outputs.
31. The Objective should specify:
32. Programme Entity Major Outputs are overall products and services delivered by FAO to specific Users, and are either an aggregation of a series of other more minor outputs to form an end-product or a single major end-product. SOFA or a fully-developed and tested methodology for natural resource assessment are examples of major end-products. A set of curricula, training materials, guidelines and methods for priority aspects of coastal fisheries management piloted and introduced in Small Island States is an aggregation of outputs. A major output is not open-ended, and is completed by the end of a TP or probably each biennium in a CP. In a TP, the major output can be the same as a "biennial output" (see below) in the year in which it is finalised, but this is unlikely. In a CP, the major output and the biennial output are more likely to be the same thing (e.g. SOFA).
33. The major output is a product, service or combination of the two. It is produced by FAO. When it is taken and used by someone else outside FAO (i.e. its "Users"), it has an immediate Outcome. The Outcome will specify what the major output is to be used for, with what expected immediate result. The scope in terms of such factors as numbers of Users who will be applying it to reach what size of audience (targets) should also be specified.
34. Users are the direct recipients of the major output, who will put it to work to achieve the outcome. They are people (engineers, policy economists, etc.) in institutions (government departments, donors, universities, etc.). The major output must be able to reach directly its intended Users. If a gap exists in reaching the intended Users, it must be addressed in the major output.
35. The geographic focus refers to where the major output will contribute to the programme entity objectives. This will normally, but not always, be the same place as it has its Outcome and where the Users are located.
36. In addition to geographic focus, major outputs can be coded by types of country within the category "All Developing Regions", e.g.: least developed; small island; and low-income, food deficit.
37. Biennial Outputs are listed in the Programme of Work and Budget. They contribute to a single major output, and are distinct products and services or groups of similar products and services, produced in a biennium. Normally, they will be products for external use, but sometimes the significant internal progress of the biennium towards delivery of a major output in a subsequent biennium may be listed, as may be a significant product for internal FAO use.
38. Indicators are requested on the Outcomes of major outputs and the programme entity Objective. An indicator provides tangible evidence of achievement of the outcome or objective and is generally a variable which can be objectively verified (measured or assessed) to determine the extent to which the Outcome or Objective is achieved, and must refer directly to it, i.e. demonstrating the causality between FAO's action and the Outcome or Objective. The Indicator is not, however, a restatement of the Outcome or Objective, and it is not a target. Indicators can be both quantitative and qualitative.
39. For an Indicator to be meaningful, it is necessary to specify (in addition to the Indicator itself) how verification will be done. This must be practicable, but may require some resources (e.g. for survey or assessing references made to information).