3.1 Background
3.2 Outputs
3.3 The ICER on ISNAR's Impact
3.4 Impact
3.5 Conclusions
It is useful to distinguish between outputs, outcomes, and the ultimate impact of ISNAR's work. ISNAR's outputs include research results (or contributions to knowledge), advice given to specific clients, management tools designed, training courses offered, and publications. The ultimate impact of ISNAR's activities, as part of a network of international centers under the umbrella of the CGIAR, is to contribute to sustainable agriculture for food security in developing countries through activities to strengthen the national agricultural research capabilities of developing countries. ISNAR's outputs lead to the achievement of the CGIAR's mission only indirectly. ISNAR's outputs lead to a number of outcomes (which are sometimes called intermediate impacts) which, in turn, can be expected to influence the achievement of the CGIAR mission - but, the relationship between intermediate impacts and ultimate impacts is difficult to measure. These outcomes relate to aspects of national agricultural research capabilities in developing countries, e.g., more effective research policies, strengthened research institutions and national research systems, individuals with better management skills, and a greater understanding of agricultural research management concerns.
The Panel has made this distinction between intermediate and ultimate impact because keeping the desired ultimate impact visible at all times provides an institution with a clear perspective on the raison d'être of its work - a perspective that can guide the setting of priorities and the making of choices. This is particularly important when institutional niche and purpose are being examined, as in this review. As noted above, the Panel recognizes that assessment of ultimate impact is difficult.
The Panel was able to gather information about ISNAR's outputs. These are provided in detail in Annex A and summarized in Section 3.2 below. It was also able to gather and analyze some information about ISNAR's intermediate impact. The information on ISNAR's intermediate impact is not as robust as the Panel would like. This is partly because the subject ISNAR deals with is not as conducive to assessment of impact as other agricultural research subjects, such as commodity research. The Panel's analysis of ISNAR's intermediate impact (for simplicity called "impact" in the rest of the report) is presented in Section 3.4.
ISNAR has two major programs. The first, the Research Policy and System Development Program, has a 1996 budget of USD 3.3 million and is the larger of the two. The second, the Management Program, has a 1996 budget of USD 2.8 million.
The Research Policy and System Development Program has organized its activities around three themes:
· policy development;
· research system development;
· new challenges.
The activities of the Management Program are also organized around three themes:
· managing research programs;
· managing resources for research;
· managing research organizations.
The activities of these programs are described and assessed in Annex A. What follows is a summary of the major work conducted by these two programs.
The Research Policy and System Development Program. ISNAR has targeted this program's activities towards senior and high-level policymakers concerned with agricultural research. The thrust of the program on the whole is couched within the evolving environment in which research is carried out. In this respect, the first two themes cover the diagnostic and planning phases of CID activities. Work in the three themes covers the following sub-themes:
Policy Development· linking research with national policies;
· resources for research, including information;
· priority setting and impact assessment.Research System Development
· strategic planning;
· system design;
· transnational cooperation.New Challenges
· biotechnology;
· natural resource management;
· agro-industries and market orientation.
Outputs under the Policy Development theme include the following:
· Linking research with national policies. ISNAR has 11 projects under this sub-theme. The most important research project is on structural adjustment. This study looked at six "early adjusting" countries, including Burkina Faso, Chile, Ghana, Indonesia, Mexico, and Sri Lanka. The study found that structural adjustment did not automatically lead to spending cuts on research, but rather that the pressure was on increasing the accountability of the research system and its effectiveness as a development tool. Several publications resulted from this work, including a book published jointly with the World Bank. Several workshops and policy-type seminar activities were also completed.· Resources for research. Work in this area relates to the redressing of the slowdown in public agricultural research expenditures and access to knowledge. The important projects under this sub-theme are the Indicator Series and the research-financing projects. The Indicator Series has been on-going since the mid-1980s. The initial data collection effort was focused on sub-Saharan Africa, and the effort has now shifted to Latin America and the Caribbean. This work has led to several important publications containing useful data and quantitative analyses. The most important project under the financing of research work is the current compilation of a Research Finance Sourcebook.
· Priority setting and impact assessment. This area covers 11 projects in various parts of the world, including Bangladesh, Benin, and Kenya, as well as some regional support projects for Eastern and Southern Africa. The target is basically to institutionalize priority-setting tools that assist in better allocation of research funds.
Outputs under the System Development theme include the following:
· Strategic planning projects are country based, and in 1996 a total of 10 projects were underway in several countries: Bangladesh, Burkina Faso, Egypt, Eritrea, Lebanon, Lesotho, Mozambique, Namibia, and Togo. In most cases, the projects aim at developing a strategic plan for the research system.· System design work falls into two main categories: organizational structure and governance issues and linkages, the latter covering linkages with farmers and with universities.
· Transnational cooperation aims to foster international spillovers at the regional level. As an example, a group of four countries pursues this objective by coordinating research on management issues (Burkina Faso, Mali, Niger, and Togo.) Each of the four countries in the group coordinates a theme, and under this arrangement Burkina Faso coordinates planning, monitoring, and evaluation; Mali, information; Niger, physical resources management; and Togo, technology transfer.
The New Challenges theme is the most recent addition to the program. Outputs include the following:
· Biotechnology. A total of seven projects are carried out under the Intermediary Biotechnology Service (IBS), all in support of new or potentially new users in Africa, Asia, WANA, and Latin America.· Natural resource management (NRM) work aims at assisting NARS policymakers to clarify the NRM issues at a conceptual level, provide scientific measures of the importance of the problem or opportunity, and explain the consequences of alternative actions.
· Agro-industries and market orientation is the most recent addition to the program, with only one project. The objective of this activity is to capture the broadening research agenda as influenced by developments in those industries that process agricultural products and under national and international market conditions.
The Management Program. ISNAR's Management Program (MP) contributes to the ISNAR goal by focusing specifically on NAROs in developing countries and helping strengthen their management capabilities. The Program sees its primary clients as individuals playing key management roles in NAROs, such as directors, managers, administrators, program leaders and scientists with managerial responsibilities. The program is organized around the following three themes:
1. Managing research programs aims to enhance the effectiveness, relevance, and quality of research programs and projects through the development of tools and approaches and the adaptation of existing management tools.2. Managing resources for research is geared towards sustained improvements in the management of human, financial, and physical resources in NAROs.
3. Managing research organizations aims to generate sustained improvements in NAROs' organizational performance.
The Managing research programs theme includes the following activities:
· The planning, monitoring and evaluation project focuses on the LAC region. It aims to analyze the region's experiences with PM&E, sensitize top-level managers, prepare practical guidelines on PM&E, introduce and disseminate PM&E methods in the region. A sourcebook on monitoring and evaluation has been published in Spanish and English as well as a baseline study on PM&E in 13 organizations. Programs to strengthen PM&E are being implemented in Costa Rica, Cuba, Panama, and Venezuela. The approach has a strong training component.· In Morocco, ISNAR and INRA Morocco jointly developed a program-planning method (programmation par objectif, PPO) which aimed at creating a methodology for planning applied research activities.
· Management information systems. This activity is aimed at developing a computer-based management information system that provides agricultural research managers with the facts they need for priority setting, budgeting, monitoring, and evaluation. INFORM is the product ISNAR has developed for this purpose. The system covers information on inputs to research (people, finances and physical resources), the research program, and research outputs (improved technologies and reports on experiments and impacts). Twenty-six countries are known to have made some use of the system to a greater or lesser degree. An intense training program (54 weeks over the last five years) has been associated with INFORM, and in this respect ISNAR plans to collaborate with the National Academy of Agricultural Research Management of Hyderabad, India. ISNAR has developed a new version (called INFORM-R) in collaboration with NAROs, based on the lessons learned.
Managing resources for research. Six projects were in implementation or completed over the last two years, while a number of others are at various stages of planning. The activities include the following:
· human resource assessment and development for ASARECA, in Guinea, and gender-related issues (4 projects);· assessing researcher performance and linking performance evaluation to the reward system (a sourcebook);
· a manual and training package on personnel management;
· an integrated model for a planning and budgeting system;
· financial resources (2 projects);
· strategies and approaches for fund raising and alternative funding mechanisms;
· a manual on the design of research facilities (produced in 1993).
Managing research organizations. This is an area where ISNAR has had a steady stream of outputs. Over the last two years, 41 projects have been associated with this theme. These include the following:
· Comprehensive institutional development (CID). CID is an approach ISNAR uses to assist a limited number of NARS in a holistic way through a wide range of activities in order to improve the performance of the NARO. The key objectives are to develop effective and efficient NARS and to improve the understanding of institutional development. The Uganda and Ecuador CIDs both focus on a number of components to be strengthened. In Uganda these included information management, monitoring and evaluation, training needs assessment, and developing appropriate financial management procedures. In Ecuador they were the design of a management information system, PM&E, human resource development and planning, transfer of technology, and extension linkages. These CID projects are jointly carried out with staff from the Policy and Systems Development Program.· Managing Change (4 projects). Preliminary work was done using an in-depth cross-country analysis of NARS a few years ago. This covered synthesis work in three clusters of countries: French-speaking countries, English-speaking countries, and Spanish-speaking countries. The results were issued as ISNAR Working Papers; Busch and Bingen utilized the materials for their publications on "Organization and Structure of National Agriculture Research" and "Restructuring Agricultural Research: Some Lessons from Experience".4 There are also projects on organizational performance assessment and management of change.
4 Discussion paper N° 93-20 (September 1993) and Briefing Paper N° 13 (May 1994).
Detailed information about the publications of staff in both programs, as well as data on training conducted are covered in Annexes A-1, A-2, and A-3.
In preparing for this EPMR, ISNAR commissioned a review of its achievements, constraints and impact to an independent team of outside consultants. The study constituted the first phase of a seven-phase project proposal on assessment of organizational performance and institutional impact in agricultural research. The team was headed by Prof. Ronald Mackay of Concordia University in Canada. The study consisted of four components: a survey of NARS leaders, a survey of ISNAR stakeholders, three country case studies (Kenya, Morocco and Uruguay), and a review of ISNAR's internally commissioned external reviews. The conceptual framework and methodology for the study was developed by the study team in conjunction with a group of ISNAR staff. The team produced a synthesis report on its findings, as well as background papers on each of the four components.
The reports contain a wealth of information on ISNAR's outputs, the perceptions and assessments of NARS leaders and stakeholders regarding ISNAR's activities and reputation, and case studies on three countries where ISNAR has had a broad-base approach to strengthening national research capabilities. Overall, the findings of the study team are positive: ISNAR has made significant achievements, its work is of good quality, and NARS and stakeholders generally report favorably about what ISNAR is and does. While recognizing that there are major methodological problems in a study of this nature, and that ISNAR's task is complex and difficult, the report concludes:
ISNAR has an important role to play in strengthening the management of agricultural research internationally. ISNAR carries out this role with vigor, resulting in a high level of client satisfaction and significant impacts on NARS and their constituent organizations. ISNAR has earned a good reputation for itself in the process.
The ICER provides a representative set of opinions of ISNAR's clients and stakeholders about the effectiveness of ISNAR. The externality of the study team members increases the objectivity of what is presented. The data-gathering effort was based on a conceptual framework of institutional assessment. The interviews were conducted very professionally, as observed by some of the respondents. In addition, the surveys provided information, heretofore not systematically available, on client and stakeholder views on ISNAR's work. Ratings by NARS leaders of ISNAR's contributions in specific areas were particularly helpful to the Panel.
The EPMR Panel discussed the methodology, approach, and findings of this study and has made selective references to the information provided in the ICER throughout the review report. It regards the study as an experiment in assessing organizational performance and one that should be used as a stepping stone for building, in due course, more rigorous tools for institutional evaluation.
The Panel identified several areas where the approach used had methodological limitations. These relate to measurement of performance, measurement of change in performance, measurement of the ISNAR-related intervention, and attribution of change to the ISNAR intervention. The indicators of performance used in the surveys were necessarily all subjective, given the methodology. Also, the methodology used did not allow control of factors other than the ISNAR intervention influencing NARO performance.
The Panel is fully aware (from its own first-hand experience during this review) that conducting assessments of organizational performance is fraught with many methodological difficulties. Perfection in this area is extremely costly, but because ISNAR's work in this area will serve as an example to others, the Panel urges the center to insist on greater methodological rigor during the next phases of this study.
3.4.1 Raising awareness
3.4.2 Influencing policy
3.4.3 Developing institutions
3.4.4 Introducing tools
3.4.5 Training
The Panel has grouped the information on ISNAR's impact into the following categories:
· raising awareness;
· influencing policy;
· developing institutions;
· introducing tools;
· training.
The following is a synopsis of the Panel's views on these major areas. Annex A includes detailed statements of the assessment of ISNAR's programs and activities.
ISNAR has made a major contribution in raising awareness about policy, planning, organization, and management issues and putting them on the agenda of research institutions, particularly national institutions.
ISNAR is also well known and appreciated for its advocacy role for the cause of NARS in the international research and development community. NARS also greatly appreciate ISNAR's role as a good and honest broker, as well as a facilitator of collaboration and promotion of interaction among NARS and between NARS and the broader research community.
ISNAR influences policy through its policy-related research, its advice to clients on policy matters, its information products of value to those involved with the analysis or making of policy, and by its training, workshops, and seminars in policy-related areas. The ICER on impact notes that two-thirds of the respondents to the NARS survey considered ISNAR to have had a positive impact in the area of formulating of agricultural research policy and named one or more examples of specific contributions. (These were classified as "adopters" in the ICER.) We comment below on the most visible signs of impact by ISNAR in the policy area.
The Panel notes that ISNAR has published a number of studies in the policy area that are considered by many observers to be of high quality. These include the Agricultural Research Policy publication of 1991 and the Science under Scarcity book of 1996. The Financing Sourcebook, now being finalized, promises to be a major publication. The Indicator Series has generated a unique set of information on the funding, staffing, and organization of national agricultural research. The ICER on the Indicator Series alludes to the positive contributions of this project, as well as potential benefit that may follow a broadening of focus as well as more extensive analysis and interpretation of data. Other notable publications are listed in Annex A. ISNAR's policy publications have a large audience within the international development community.
ISNAR's work on the policy implications of biotechnology (through the Intermediary Biotechnology Service) is a highly regarded activity. A mid-term evaluation of IBS conducted in 1995 provides useful insights. The evaluators concluded that IBS had impressive achievements in terms of activities (interactive policy seminars) and outputs (database and technical publications) during its two-and-a-half years of implementation. Furthermore, IBS has developed the capability to consolidate its activities and output into potentially useful decision-making tools applicable to biotechnology management. IBS has also successfully established its visibility among potential clients, which include developing countries as end users and international biotechnology programs as intermediary users.
ISNAR has a few "flagship" cases of institutional development during the period under study. Prominent among these are work with the NAROs in Kenya, Morocco, Uganda, and Uruguay. In each of these cases, ISNAR has had sustained involvement over several (in some cases, many) years with the NARO. Interventions have been jointly designed by the client NARO.
An ICER is available on the Uganda work. ISNAR's interventions in this country started in 1986. The panel that conducted the ICER gives ISNAR strong marks for its contributions through 1994. Interventions during the last two years have ranged from development of a research management system for the NARO to installation of management information systems (INFORM) and work to strengthen monitoring and evaluation practices. The ICER gives ISNAR mixed marks in terms of generating change during this latter period (p. 12, Uganda ICER):
...the program components included under the CID approach form a sound basis for strengthening the holistic development of NARO and its associated institutes, thus enabling them to plan, implement and follow up their activities in a coordinated manner. However, despite this overall positive outlook, problems have been encountered during implementation. Several components/sub-components have either not been implemented or implemented poorly. Failures are due to neither NARO nor ISNAR alone. It is indicated by NARO management and staff that a higher level of success could have been achieved if program implementation had been started with adequate preparation in terms of availability of adequate staff, funds and facilities.
There are also three case studies (Kenya, Morocco, and Uruguay), conducted by a member of the team that conducted the impact assessment ICER, that shed light on the outcomes of ISNAR's CID work. These case studies illustrate that ISNAR's involvement with the NARO generally helped clarify research policies at the governmental level, in addition to introducing changes in the organizational or management processes of the NARO.
The Morocco case study notes significant impact from ISNAR's work in the NARO: "This case study found that ISNAR has had and continues to have a substantial impact on INRA, the principal public NARO in Morocco. This impact is evident in changes and improvements in the latter's environment, motivation, capacity and performance to varying degrees" (p. 5). The Kenya case study recognizes the importance of close partnerships for the success of the intervention.
Over the years, ISNAR has developed or adapted a number of management tools for use in agricultural research organizations. The most prominent among these is INFORM, a management information system. There are also tools or approaches in strategic planning, human resource management, monitoring and evaluation, and other management subjects ISNAR has promoted with its clients.
The ICER on impact provides information on the views of NARS leaders on ISNAR's contribution to the use of several management tools or processes within the NARS. The areas receiving the highest ratings include human resource management and strategic and long-term planning (76% and 73% of the respondents, respectively, noted positive contributions by ISNAR and named one or more specific examples). The lowest percentages were for financial management and accountability (37%) and organizational culture and staff commitment and loyalty (33%).
The Panel comments below on indicators of ISNAR's impact in four areas: priority setting, INFORM, PM&E in Latin America and the Caribbean, program planning in Morocco. ICERs are available on INFORM and PM&E in LAC.
Priority setting. In the area of priority setting, ISNAR has recorded its greatest impact in a number of countries, including Benin, Burkina Faso, Kenya, Mali, Morocco, and Uganda. Some of the recorded lessons include the importance of priority setting as an internal process, and that the process has to be consistent with organizational capacity and resource base, including human resources, if these tools are to be widely used. Few priority-setting exercises appear to have led to verifiable changes in resource allocation because the activity is de-linked from the resource allocation activity. ISNAR has apparently experienced that, with the exception of a few cases, the priority-setting exercises they support have insignificant legitimacy in terms of local influence on actual priorities or research agendas. There is yet no evidence of how sustainable the priority-setting techniques are after initial adoption.
ISNAR has assisted both SACCAR and ASARECA with priority-setting efforts at a regional level and this is expected to have a positive impact on national systems.
INFORM. The ICER on INFORM was completed two years ago. At the time it had been implemented in 16 countries. The ICER concluded as follows: "Results have been mixed. Some countries are making good progress in implementation of INFORM; yet none can be said to have fully operationalized it through its integration into the programming, monitoring and evaluation cycles" (p. 1). After describing the problems hindering more rapid progress, the ICER notes that "a key weakness of INFORM is that the databases do not include information that can be used in the assessment of the quality of the research nor the usefulness of research findings" and makes several recommendations for improvement.
ISNAR has made several improvements in INFORM over the last two years. A new version was tested in Zambia and a French version is in preparation. There are several signs of its use by others. For example, a UK consultancy company assisting Ghana's National Agricultural Research Council has chosen INFORM as the tool for the management information system. Similar use is made of INFORM by Winrock International in a project in Pakistan. In India, NAARM adopted INFORM for use in its MIS training for ICAR in 1990. Now, NAARM is interested in the new version of INFORM.
PM&E. The LAC project was evaluated in 1994 and given excellent marks in terms of both its outputs and the participatory process used in executing it. The outputs include over 100 managers trained, training materials on PM&E, a sourcebook on M&E (in Spanish and English), 18 persons trained as trainers in PM&E, and case studies on the status of PM&E in 13 organizations. These are all valued outputs, and surveys of NARS and stakeholders have singled out this activity as one of the most useful carried out by ISNAR.
The Panel considers the 13 case studies as very important in measuring change over time in PM&E capacity in organizations. By designing the project to include baseline data collection, ISNAR is demonstrating how before-after measurements can be used to monitor and assess change. The Panel notes that had a similar approach been used by ISNAR in relation to its own interventions over the years, baseline data would have enabled ISNAR to assess change attributable to the center's efforts more objectively (and facilitate the work of external reviewers like us).
Program planning. The Morocco project was developed at the request of INRA as a novel approach for planning applied research activities for commodity-based and production-system-based research programs. Staff from INRA-Morocco participated closely with ISNAR staff in the design of the approach. The planning tool is used by INRA and other NAROs. Since 1990, 12 out of INRA's 18 research programs have been planned using this method. A number of other countries, including Benin, Mali, Senegal, and Tunisia, have begun or are beginning to apply this approach to planning.
ISNAR training refers to activities whose principal objective is to improve the capacity of professionals. This activity includes group training, workshops, and seminars. The ICER on impact notes that ISNAR has delivered over 100 management training events internationally, reaching over 2,500 agricultural research managers. According to the ICER: "NARS leaders, in response to a question asking them to identify specific products and services that ISNAR has provided to their organizations, cited training and human resource development most frequently" (p. 15). This reflects the visibility of training as an ISNAR activity by the NARS leaders.
The Panel concludes that, during the review period, ISNAR has produced a significant number of outputs, and that most of these have been appreciated by its clients and stakeholders.
ISNAR has had some positive impact on national agricultural research capabilities in several developing countries. These are in the areas of raising awareness about research policy and management issues, influencing research policies in developing countries, helping develop research systems and institutions, introducing useful management tools, and in training. Although the information available to the Panel does not allow it to assess the extent of ISNAR's ultimate impact, the Panel's impression is that ISNAR's overall impact during the review period was more significant than during the period covered by the last EPMR.
There is more concrete evidence of the specific impact of some of ISNAR's activities or projects, some of which has been noted above. ISNAR's impact has not been evenly sustained across activities. The Panel's impression is that activities with strong participatory involvement of clients, carried out in the "action research" mode, and couched within the perspective of long-term comprehensive institutional development have had longer-lasting impact. Integrating baseline data collection in all new projects and progressively accumulating and/or synthesizing lessons across projects would help build a stronger information base for assessing impact in future years.
