8.1 ILRI's Approach to Research on Assessment of Impact and Policy Analysis
8.2 A Methodological Consideration
8.3 Achievements in Impact Assessment and Policy Analysis Research
8.4 Assessment of Quality of Research
8.5 Adjusting the Strategy for a Stronger Link between Impact Assessment and Policy Analysis
This chapter deals with research regarding the assessment of impact of ILRI's projects and the policy research needed to extend the benefits of ILRI's output. It does not analyse the impact of ILRI in aggregate, an issue addressed in section 11.2.
Although Project 11 (Systems Analysis and Impact Assessment) was discussed in chapter 7, because of its relation with sustainable production systems, it is also reviewed here because of its potential relation with project 12 (Policy Analysis). Project 11 evaluates the impact of the other ILRI research projects. Its objectives are to provide estimates of impact (ex-post analysis) of the knowledge, processes, products and technologies developed by ILRI and the expected impact (ex-ante analysis) when adoption occurs. The project has an annual budget of US$ 1.6 million, but will diminish to US$ 1.3 million by year 2000.
Recognising the importance of an adequate policy framework to create the desirable conditions for technological innovation, ILRI has implemented a project on policy analysis (Project 12). Its objective is to identify and quantify the impact of livestock policy and institutional reform on livestock productivity and technology uptake. The project has an annual budget of US$ 1.65 million, for three years. The analyses to date focused on policies to favour the extension of small holder dairy cattle production systems, better animal nutrition and animal disease prevention and control, and balancing livestock productivity and environmental sustainability.
Projects 11 and 12 are undertaken with limited interaction between them. Yet, as it will be discussed here, the need for alignment of their agendas and for complementary efforts prompted their analysis in this chapter.
For a proper assessment of impact of ILRI projects, it should be recognised that research conducted by ILRI generates outputs that have value themselves, before reaching farmers. The output can take multiple forms, including a vaccine, a genetic resource, a process, the design of equipment, technologies, and knowledge of prevailing conditions generated by a diagnostic. Products of research, which also have value are publications, including training manuals, journal articles and books which increase awareness and knowledge. Therefore, the output of livestock research has to be assessed in the context of its potential impact, and there is need to understand and monitor the additional steps until impact is achieved at the level of ultimate users.
When there is an uptake of research output, the process generates outcomes. These take forms including: commercial production of seeds and equipment, changes in technologies, farm level investments, production of breeding cattle and changes in farming systems. Outcomes are therefore intermediate stages in the way to achieve impact. Even when pursued by the research organization, successful adoption often requires policies that encourage the transformation of research products into assets for development through improved income and quality of the environment.
The output of research and its eventual impact occur in many ways. For example, when a vaccine is transformed into a commercial product, and is used by producers in their cattle, the mortality rate diminishes and productivity increases, hence it has positive impact on income. When germplasm becomes a commercial seed used by producers, it may result in increased yields, less use of fertilisers and water and better profits. Impact can also result when a design is patented and thousands of units are produced, purchased by farmers, and used, saves power and relieves the farmer from physical effort, increases efficiency, lowers cost and increases net income. Impact also takes place when, aware of the results of a -diagnostic, governments launch a programme to control a disease, which reduces cattle mortality, improves their strength and working capacity and prevents the decline in the stock of animals, a most important asset for farmers.
In all these examples there is need for policies and investments that reduce transaction costs, and increase the prospects for positive impact. The extent of outreach to large numbers of beneficiaries and the amount of positive change depend on the effectiveness of policies. Such policies, therefore, are aimed at creating conditions for research outputs to become outcomes that have impact and contribute to development. Understood in this way, the process can lead to a better approach to assess impact and analyse the effect of policies.
Projects 11 and 12 have generated output intended to assist ILRI in revealing the returns to investment in livestock research, and potential gains to be derived by adequate policies. The outputs of this research, although interesting, deserve further discussion in light of basic assumptions on which the studies were based.
Valuable contributions were made by Project 11 regarding methodologies for impact assessment. Also the Project has addressed the problem of measuring environmental effects of technological innovations and contributed analyses of environmental improvement at farm level in the Highlands, the Sahel and southern Africa
An ILRI study by Kristjanson and co-workers measured costs and returns to control of African animal trypanosomosis, and was also discussed in Chapter 7. The Study presented the potential economic benefits of a vaccine discovery, estimating a 33% internal rate of return and 34:1 benefit-to-cost ratio. These results should be taken with much care as they could mislead decisions on research priorities and allocation of resources. Assumptions about the time required to develop a vaccine may have been optimistic. Also, the analysis assumed no transaction costs between vaccine delivery and actual adoption by farmers. A systematic evaluation of all assumptions regarding the feasibility of output in all ILRI's research projects becomes a most justified task.
Early in 1996 ILRI performed an analysis of expected returns to its research in four major areas: feeds and nutrition, genetic resistance to disease, vaccines and dairy production technology. The ex-ante assessment was done using an economic surplus model on the basis of several assumptions regarding the number of years to output, probability of success of the research, rate of adoption by farmers and adoption lag. The analysis shows that the highest rates of return are expected in developing disease resistance to helminths (42 percent) and developing vaccines (30-35 percent). Changes in the assumptions and accounting for transaction costs may have some effects in the results, yet such sensitivity analysis was not reported.
Recently, Project 11 published a Compendium of ILRI Research Impacts & Adoption for 53 projects, executed between 1975 and 1998. The data were provided by Project Co-ordinators, but were not in all cases the results of an impact analysis, but rather were assertions of Project Co-ordinators. Almost 50 percent of the reports refer to projects already completed, the remainder refer to on-going projects that had already generated some output. Also, 50 percent of the cases reported were ex-post analyses, the remainder were ex-ante studies. The summarised information reveals that ILRI work has generated valuable outputs. However, the outcomes and impact are not always presented, nor can they be determined solely from the available information. The potential is there to demonstrate that some of ILRI's work was worthwhile; but needs some conceptual and operational clarifications and some additional work to get homogeneous information for all cases.
Project 12 (Policy Analysis) provided valuable results of research on improving soil, water and nutrient management: option to increase market efficiency and competitiveness; and institutional reforms for efficient delivery of animal health services. For the several cases analysed, the Project has relied on field and market data, detailed specification of the current policy framework, and blending of technological and economic coefficients. Also it has used quantitative methods and models to capture behavioural relations and interactions. The latter include statistical and econometric models, simulation models and a Policy Analysis Matrix. Most valuable has been the partnership with national organizations responsible for policy and with IFPRI.
The analysis has shown the importance of adequate policies to remove institutional and market constraints. Such are the cases of economic and social benefits due to: i) increased number of N'Dama cattle, resistant to Trypanosomosis, which allows higher production of milk and meat as well as growth of the number and quality of animal stock, ii) provision of land tiding as a necessary condition to undertake farm level investments, utilisation of manure to fertilise fields and adoption of technological innovations, and iii) organization of producers through co-operatives as a means to reduce marketing costs for milk produced in small households. These are a few examples of the relevant issues addressed.
The research in Project 12 is done in partnership with national and international organizations. The co-operation with IFPRI has proved to be most useful on a bilateral basis and within the Systemwide Property Rights Policy Research Programme convened by IFPRI.
Project 12 developed novel means for outreach, to let others know about the importance of adequate policies. In this regard, there are numerous papers published in refereed journals and internal ILRI publications. The Policy Briefs and the Policy Dialog are an effective means to reach key people, who do not usually have the time to read long reports, books and journal articles, yet they can provide a quicker response regarding policy decisions. The scientific and refereed publications have a different, desirable audience, but are usually not suited for policymakers. Other publications like the Livestock Policy Analysis Manual were also used for training.
An appraisal of ILRI's capacity for research on impact assessment and policy analysis reveals that it ranks well in the Institute's portfolio. These two projects together comprise a group of highly qualified professionals, with experience in Africa and employing sound methodologies. The quality of the research is reasonably high, nevertheless a refinement is needed to present results in a manner more usable for decision-making purposes, at ILRI and elsewhere.
The projects have focused on specific problems, which allows for in-depth analysis; however, they are challenged to cover a wider array of issues that deserve policy decisions to facilitate innovations in the livestock sector. Rigorous quantitative methodologies are an indicator of quality of research, yet their use is usually data- and time-demanding. Some sacrifice in quantitative analysis may be needed in order to address more issues rapidly. A multidisciplinary, highly participatory effort may provide an alternative to large data collection quantitative approaches.
These two projects are important in assisting ILRI to gain strength and recognition. They provide elements of judgement for the relevance and utility of research outputs. Their own output is a key resource to assist ILRI in developing a strategy for partnerships with influential organizations in the countries, and for the definition and implementation of development policies. They include the national authorities at the Ministries of Agriculture and Livestock, Economics, and Trade, as well as those responsible for managing national budgets. On the international side there are institutions such as IFPRI, FAO, the World Bank, and the regional development banks, with which there is already some ongoing co-operation.
Projects 11 and 12 interact with other projects in the SPS Programme, yet more could be gained from some adjustments. First, such interactions should begin at the stage of design, to assure the identification of policies that reduce the constraints for the adoption of the project output. Second, stronger relations should also be developed with projects in the Biosciences Programme. Third, the staff in these projects, now operating in the Nairobi and Addis Ababa campuses should be placed together at one location.
The impact of ILRI work needs to be assessed not only on the basis of the contributions of research, but also on the other activities that the Institution performs. Scientific and technical aspects of research include the main activities (amounting to 77 percent of the programme budget); others include those under project 20 regarding strengthening of NARS, information management and networks (14 percent of the programme budget) and assessment of impact and Policy Analysis (9 percent of the programme budget). The assessment of impact should therefore examine contributions made by all activities, particularly if, as part of its global mandate, ILRI could become more involved in those other activities complementary to research.
ILRI should assure that its research outputs have value and provide clear signals of its potential impact on income of producers and the quality of natural resources. It is also important to create increased awareness that the potential impact at farm level may be limited by existing constraints, which can be removed by policies and investments. ILRI should therefore consider the high return to improving the assessment of its work, revealing the potential impact of research outputs and contributing to sound livestock policies. In this way it can assist governments to create adequate conditions, institutional reforms and investments, including those that create greater research and extension capacity.
To enable the necessary integration of impact assessment and policy research, better orient the Institute's biophysical and production systems research (and its priorities), and provide a firm base for delivering outputs and generating impact, the Panel recommends that Projects 11 (Systems Analysis and Impact Assessment) and 12 (Policy Analysis) be merged, with all staff operating at ILRI's headquarters in Nairobi.
