11.1 Assuring the Quality of Science
11.2 The Impact of ILRI
11.3 Globalisation of ILRI's Agenda
11.4 Intellectual Property Rights
11.6 Bioethics and Animal Welfare
This chapter covers matters that affect the Institute and its activities as a whole and therefore required analysis and comment by the Panel. Some are matters of policy or international concern that can and do affect the way ILRI does its work. The subjects covered are: assuring science quality, impact of ILRI's research, globalisation, intellectual property rights, biosafety and bioethics and animal welfare.
11.1.1 Good Science and Its Assessment
11.1.2 ILRI's Approach to Ensuring Scientific Quality
What is "good science" - can we recognise it, and can it be evaluated? These questions are asked in almost every research institution, and are especially pertinent for ILRI because of its range of research activities from basic or strategic to applied, and its wide range of scientific disciplines. The Panel asked, what should be expected from strategic or applied livestock research and how can its scientific quality be assessed? To help set the stage, concepts and principles were reviewed that could be important in assuring scientific quality.
In his insightful book. The Joy of Science: Excellence and its Rewards. Sindermann (1985) defined 'good science' as: " ....production of extensive data, rational insightful application of those data to hypotheses about natural events, and effective presentation of the resulting information and analysis to colleagues". This definition aims at individual scientists or teams of scientists with clear objectives, and deals not only with conduct of research but also in communicating results to others. Hence, publication of research results is an essential part of research and is important in evaluating scientific quality.
Sindermann went further in defining "good science" by proposing three categories along with some of their characteristics:
"good science " - 'sustained research productivity in a chosen area of subject matter, supported by substantive papers in major journals.
"very good science " - a series of major research papers over a period of years, which in the aggregate provide a substantial addition to knowledge in an area of subject matter, or definitive and creative reviews in a specific subject, or specialised books in an areas of expertise.
"excellent science" - could include a brilliant series of definitive research papers, exploring in depth a previously little-known phenomenon; a significant conceptual advance; a masterful elegant synthesis of the disparate data of others; or a definitive, award-winning textbook in a major scientific field.
Most science is carried out in organizations and institutions, therefore the way an institution plans and manages research has much to do with scientific quality. A research institution must develop a clear vision for itself and its work, set priorities, and define strategies to carry out the work. If these are done well, excellent staff will be attracted and outstanding staff will be retained.
Institutional strength or core competence in science has been defined: "A foundation of excellence in the practice of science constitutes the core of success, around which additional interpersonal embellishments may be added, but which can never replace the core " (Sindermann, 1985). This definition appears to include the concept of 'critical mass'; the Panel notes that ILRI has encapsulated this idea somewhat in its 'platforms of essential capacity'.
Most scientists are happiest in a creative institution that is 'going places', that knows what it is doing and where it wants to go, and is likely to get there. Professional satisfaction is a powerful motivator, and morale of scientists often depends on whether or not their work is appreciated.
Some common methods used to measure and assure scientific quality include at least some of the topics below:
- Publication record (including quality of journals in which work is published)
- Citation analysis - a good way to determine how important or useful a publication is considered by other scientists.
- Peer review - an essential part of any scientific enterprise, and a check on ideas, value of research, and scientific recognition and affirmation.
- National or international recognition and awards - a way to measure where an individual scientist rates in his/her professional discipline in regard to scientific competence, leadership, and respect
- Impact of the work on science or society - Impact is very difficult to measure, but answering some questions may help: Has the work changed the way research is done? Has it produced new ideas that open new doors? Could the development lead to new processes or important products? Will it add to the body of knowledge so as to advance a field or provide new insights or solutions to societal problems? Does it produce new methodologies?
- Staff evaluation procedures
The Panel did not find it easy to understand ILRI's system to assure science quality, finding it vague. However, some elements of a nascent system could be identified and are discussed below. There is no Publications Committee, and decisions about what can be published rest with project co-ordinators and the Programme Directors. Overall research quality is the responsibility of the Programme Directors.
ILRI uses a conventional staff appraisal system that begins with objectives and plans, and is highly dependent on the experience or skills of the supervisors) to deal with oversight, questions of output and quality of performance, and matters of career development. Assessment of both national and international staff is the responsibility of the Programme Directors. There has been and will be more supervisory training.
Centre Commissioned External Reviews (CCER)
ILRI's original thinking was that CCERs: "..offer an excellent opportunity to gain [he perspective of qualified experts on the merits of recent and current programmes and to provide a reaction to proposed future directions. The term merit is intended to include both the scientific quality and the utility of the products being judged" (Agenda Brief, ILRI Board of Trustees, April 16-19, 1996).
The ILRI Board commissioned five CCERs during the review period: Production Systems Research (1995), Systems Analysis and Impact Assessment (1996), Livestock Policy Analysis (1996), Animal Health Improvement (1996), and Information Services (1997). A main purpose of these CCERs was to help ILRI during its formative years and to help prepare ILRI's first MTP. Peer evaluation of ongoing research was considered essential to link the differing cultures of the two former institutions and "to ensure quality and utility and applicability of ILRI's products in practice". Indeed, CCERs became a key part of ILRI's planning process and were asked by the Board "to review briefly progress and accomplishments, but to focus on the future plans of the area under review", and to develop "a critical external review of the plan ".
The Animal Health Improvement CCER panel commented on science quality, terming it; " ...outstanding, strategic... high quality of science.. ". The CCER on Livestock Policy Analysis commented: ".... very favourably impressed by ...quality and relevance of the research...mix of cutting-edge scientific... and... conventional approaches". The CCER on Production Systems Research stated; "..a tendency for some... work ... to be rather pedestrian, ...pay more attention to its strategic thinking, visionary approach and innovative research in PSR". The Panel noted that the methods of assessing quality were not stated clearly in these reviews, and that more attention was given to assessing ILRI's future plans.
Panel Comment: In its Terms of Reference from the TAC Chairman the Panel was asked to evaluate science quality and the role CCERs had in its assessment. The Panel regretted that the CCERs were used more for planning or reviewing plans than for assessing scientific quality, concluding that most were weak or lacking in such assessments. The Panel considers it essential that future Terms of Reference for CCERs state specifically the need to assess science quality rigorously and the assessment methods and indicators that should be used.
In preparing for the EPMR, ILRI assessed its publications from 1992-1997, and especially 1994-97. Since 1992 these totalled 1558, including 774 journal articles. Since ILRI's establishment in 1995, 741 in total were produced, 339 in refereed journals. In 1990, 42% of all CGIAR publications were in refereed journals; the figure for ILCA was 31%, and for ILRAD was 89%, the highest of all CGIAR centres. During 1992-97, 50% of ILRAD/ILCA/ILRI publications were in refereed journals.
In 1993, average CGIAR publications per scientist totalled 1.38, of which 0.58 were journal articles; corresponding figures for ILRAD/ILCA/ILRI (including graduate students) during 1992-97 were 1.45 and 0.73, respectively. However, during 1995-97 after ILRI's international scientist numbers dropped from 116 in 1992-94 to 97 (1995-97), one result was higher average rates per scientist, 2.48 and 1.23, respectively. Journal numbers per scientist before ILRI was established (1992-94) were 1.25, and after ILRI was established, 1.16.
Publication rates (1994-97) for the Programmes were: Biosciences, total 452, journal articles 296; Sustainable Production Systems, total 481, journal articles 177; SPAN, total 38, journal articles 1.
During 1992-97 ILRAD/ILCA/ILRI scientists published in 186 journals, including Nature, Genetics and Science. Thirty-nine journals account for 549 of the 774 articles published; these journals range over a wide number of fields from pastures to biotechnology and molecular parasitology.
Panel Comment: The Panel found ILRI publications difficult to evaluate, partly because of the carryover from ILRAD and ILCA, and partly because various reports on ILRI's publication record that were received by the Panel were not in agreement, e.g, in terms of inclusion non-refereed journals or not, counting the same publication more than once, and discrepancies in numbers of publications by projects.
To monitor and target ILRI's reporting of scientific findings, the Panel urges that: 1) a portfolio of priority journals be identified for the reporting of original research; 2) journal articles be catalogued by research theme (or platform), programme, and author, distinguishing past employees and outside collaborators from current staff; and 3) these procedures also be used for other publications. This would enable the Institute to describe, quantify, and monitor the quality and flows of information from its research programmes to selected audiences.
Annual Programme Meetings (APM)
Annual Programme Meetings are intended to provide a means for peer review and science quality assurance within ILRI, but the Panel saw no evidence of these at the September, 1998 meeting in Addis Ababa.
Panel Comment: The Panel believes the APM could be useful for peer review and for quality assurance, but that more detailed presentations and incisive discussions would be required to make them effective in this regard.
After considerable thought, the Panel surmised that one way to approach science quality at ILRI was to use (with apologies to Charles Dickens) the concepts of science past. science present and science future in particular to help to understand the influence the former ILCA and ILRAD had on ILRI and its scientific quality. For example, ILRAD was considered as doing high quality strategic (and basic) research, some of it considered not only world-class but also making new research possible (e.g., developing methods to grow trypanosomes in vitro). Many ILRAD publications were considered to be of high quality and the work was considered excellent. The Panel noted that some ILRAD research was published after ILRI was formed, many by scientists never at ILRI, but whose articles count as ILRI publications. This work could be classified as science past, in that ILRAD's products are still emerging and influencing ILRI's present work, but how do (or should) they count in assessing ILRI's science quality?
On the other hand, ILCA from its beginning was often criticised for lack of focus; further, criticisms were heard about research quality, that its research did not meet standards expected of an international centre, and that it lacked achievements or impact. A notable exception was the Trypanotolerant Livestock Network, where both research quality and publications were well regarded. The Panel concluded some of ILCA still bears on ILRI's science past, and the carryover of much of ILCA's research agenda affects ILRI's science present. Also, as with ILRAD, some former ILCA staff are no longer (or never were) at ILRI, including some with impressive research and publication records.
So now we come to ILRI's science present, which is an amalgam of two former institutes - adjusted and modified since 1995 - plus new activities, some relating to its global strategy. But how do we assess science present? How much do we go back to ILCA and ILRAD and their mandates and activities, or do we just start today as if those centres never existed? And do we ascribe science advances made - or not made - to ILRI alone? Publications cannot help much because of issues of science past. Neither are CCERs of much help, because they were used more in planning than in assessing science quality. Further, some ILRI scientists may enjoy high standing, but how much is a reflection of science past, and how much is science present? The Panel believes assessments of science quality must be done at sub-project, project, programme and Institute level, but questions how these can be done without bringing in matters of both science past and science present.
We turn now to impact analysis. Ex post assessments by their nature relate to science past. The question is, should past impacts count to ILRI, or just be counted to global livestock research in general and assigned to the legacy of ILRI and its predecessors? Ex ante assessments are estimates of potential impact and their role in quality assessment is uncertain. It could be argued that ex ante assessments are geared toward both science present and science future, and while necessary, are not sufficient to measure quality.
That brings us to science future. Can assessments of quality of science past and science present help to predict ILRI's future science quality? The Panel believes they can, if its vision, priorities, and strategies are clearly defined. Science future depends largely on how ILRI goes about its business, including developing rigorous mechanisms to ensure high quality science. A clear commitment to quality will attract and hold outstanding scientists. It is possible of course that some 'ghosts' of science past and science present will still be around to affect ILRI's science future, but it the hope of the Panel that they will be mostly positive or at least benign.
One last thought on science future at ILRI. It may be possible that the 'snapshot' our Panel had of the Institute might have overestimated (or underestimated) science quality in some projects or activities, and that some areas might be weaker (or stronger) than we thought. Therefore, the Panel tried to complement its assessment using a matrix scoring system for quality for all projects, which is discussed below.
Assessing ILRI's Science Quality Today (Science present)
The Panel is convinced that ILRI with its global mandate must know where it is going, have clear plans to focus on a few very important topics, and allow its scientists the freedom and support to use their intellect and creative talents to solve them. Because of this, the Panel believes that ILRI, by setting clear priorities and laying out appropriate strategies, will affect positively its scientific quality.
The Panel was concerned that ILRI may be too budget-driven, compared with assuring efficient use of the funds in hand, and that this may affect science quality. The constant quest for restricted funds may result in erosion of core scientific competence ILRI's 'platforms of essential capacity'.
A Matrix Scoring Approach Used bv the Panel in Its Assessment
Because some of the approaches or elements of assessment used by ILRI did not fully satisfy, a matrix scoring system was used to assist the Panel in its analysis.
To assess overall quality and output of individual projects and of each research programme, the Panel devised a simple method of matrix scoring composed of five factors. These factors were: overall scientific quality, publication quality, output, critical mass. and research focus. The scoring scale contained values ranging from 1 to 5, centred on a median score of '3'. The median score was equated to Sindermann's definition of "good science ", or "... production of extensive data, rational insightful application of those data to hypotheses about natural events, and effective presentation of the resulting information and analysis to colleagues". Larger values represented higher quality and smaller values represented less quality, and the need for managerial action.
The Panel's position was that every publication of original research by ILRI should meet or exceed the criterion of 'good science' - it is the minimum threshold of science quality for a CGIAR centre. Consequently, programme areas or research themes that fall short identify a potential need for adjustment or other action.
While this process is not entirely objective, as could have been achieved with more time and information, the Panel considered that the results provided a useful approximate overview of general trends and possible reasons for quality variations. The method was applied to each research project, and salient results were discussed briefly in the Summary sections of Chapters 6 and 7. The assessment of overall quality utilised two principal explanatory factors - research focus and critical mass. This is the sort of assessment process that Management could undertake to identify areas of strengths and weakness, and to help determine appropriate interventions to improve or maintain the quality of science.
Analysis of the resulting matrix of scores covering the 19 research projects led to the following working-level interpretations:
· The overall quality assessment showed that 60% of the projects were scored greater than or equal to very good; 28% of good quality; and about 12% unsatisfactory, indicating substantial room for improvement. Additional considerations of output combined with publication quality showed that 35% of the projects were scored equal to or better than very good, 35% were good, and 30% were unsatisfactory. Only 5% of the projects were rated as excellent.
· A key issue for improving the output and quality is to focus on a specific set of high priority objectives and to concentrate efforts on greater output and first rate quality. Some 50% of projects showed good focus, which was roughly correlated with the amount and quality of output, and highlighting the need for greater research focus. About 22% scored unsatisfactory for focus.
· The Panel considers that certain projects also lack critical mass, and this could be inhibiting output.
· The analysis suggests that most ILRI research meets acceptable criteria of good science. The research programme of an international centre should be highly visible with significant outputs of very good and excellent science across all of its projects.
· The Panel's preliminary analysis suggests a potential fragility in ILRI's science quality, given the relatively small proportion of projects with outputs or quality that rate as excellent and the proportion below minimum standard. The compounded effects from losses in key staff members, difficulties in recruitment, variations in funding and increasing dependence on restricted funding could strain certain projects and jeopardise quality and output. These factors call for appropriate early measures to improve ILRI's science quality.
The Panel's analysis indicates that ILRI has some elements that could be used to construct a mechanism for scientific quality assurance. A number of possible methods are presented in Section 11.1.1. The Panel believes a strong Publications Committee would be needed. CCERs and the Annual Programme Meetings could be used better for peer review and scientific quality assessment. ILRI also has scientists who have received recognition by professional societies or other distinctions of honour. What is needed is a clear mechanism to ensure science quality at every level in the Institute.
To maintain and enhance ILRI's scientific reputation, the Panel recommends that ILRI develop and use explicit mechanisms to ensure scientific quality and the effectiveness and utility of its outputs.
Over some 20 years ILRAD and ILCA invested close to US$ 500 million in research. Since ILRI was established, its investments added a further US$ 100 million to that figure. TAC and CGIAR have expressed concern regarding the impact of ILRI research.
The pertinent chapters in this report have reported, albeit not comprehensively, ILRI's outputs and achievements in its various projects. Also, the preceding section in this Chapter tells how the Panel went about assessing the quality of science. The Centre has not submitted a document that would attempt an aggregate appraisal of ILRI's impact. Such an assessment would have allowed ILRI to present the results of the last four years, on the basis of the investments made in the previous years. However, the analysis performed by project 11, referred to in Chapter 8, has made a major effort to shed light on impact of the Institute's work in a wide range of areas. The Panel commends ILRI for this valuable information.
Considering the analysis in the previous chapters, the conclusion of the Panel is that the direct farm level impact of ILRI is so far rather limited, although quite substantial local and regional uptake of some of the technologies produced is reported. The Panel noted particularly the considerable adoption of the fodder bank technology in West Africa and of the Vertisol management technology in Ethiopia. In 1998 the latter technology was selected as one of 40 outstanding innovative projects by the Third World Network of Scientific Organizations of UNDP.
Also, ILRI has produced a significant number of publications and journal articles. They have contributed to increased knowledge and awareness, yet it is hardly possible to evaluate their impact without a specific analysis. In this case the issue of impact on ultimate users is even more difficult to assess and information to do this is not fully available.
The output of ILRI's research is diverse, including knowledge generated by diagnostics and publications, equipment designs, processes for production, and research methods. These outputs go through several stages until they become outcomes ready for transfer and until they eventually reach ultimate users. Furthermore, outputs of research enter the stream of adoption and there may be additional interventions required, particularly for the transformation of such outputs into final inputs, products and services where transaction costs and market failures may, in addition, largely determine uptake.
ILRI is encouraged to continue its ongoing effort to assess its overall impact over the last four years as a basis for its strategic orientation.
A principal recommendation resulting from the review of international livestock research, leading to the formation of ILRI in 1994/5 was to expand CGIAR livestock research so far primarily focused on Africa to a broader portfolio which addressed priorities on a global scale. The founding documents prepared under the aegis of the Rockefeller Foundation and later adopted by the ILRI Board of Trustees in the Strategic Plan identified two kinds of system-wide responsibilities for the new Institute: (1) a global mandate for livestock research by the new Institute, and (2) a convenor role with a mandate to provide leadership in coordination and communication about livestock related research. The latter role has since been specified in the context of the System-wide Livestock Programme (SLP) for which DLRI has assumed convening responsibilities. The Panel has recommended an adjustment of ILRI's interpretation of the convenor function in Chapter 10.
The Panel wishes to stress that it strongly endorses the decision to expand the mandate of the unified Institute to the global dimension, although it realises that the funding situation since 1994, when the Institute received this Mandate, was not favourable to major deployments in this regard.
When analysing ILRI's approach to the design and implementation of its global agenda, on the other hand, the Panel had some difficulties to establish in detail how the Centre exactly interpreted its global mandate and how it intended to go about fulfilling this.
In the introduction of the global mandate, ILRI's strategic plan first considers global needs for livestock research and the relative capacities of institutes both within and outside the CGIAR to contribute to the research goals. It then identifies broad research opportunities to improve animal performance, systems productivity and livestock sector performance and technology transfer. Research opportunities of global relevance identified for ILRI were addressed in this broad context and are addressed below:
· ruminant health (vector-borne tropical diseases), development of new control technologies, epidemiological methods "
· genetics of disease tolerance "
· ecoregional systems research on feed sources, quality and utilisation
· livestock system research and impact analysis
· natural resource management, contribution of livestock
Since 1995, ILRI has conducted a series of consultations with relevant institutions outside Africa to help establish priorities within the broadened agenda for livestock research. This coincided with the inception of the System-wide Livestock Programme for which ILRI accepted convenor responsibilities.
For the current MTP (1998-2000), ILRI addresses its expanded global mandate in two ways:
· by expanding recommendation domains for strategic research outside Africa
· by placing multidisciplinary teams in Asia, LAC and WANA
ILRI's research output is designed to generate international public goods with relevance beyond the boundaries of the location or region in which they are generated. Such strategic research, for which valid recommendation domains should be feasible, are the areas of parasite biology, bovine immunology, genetics of disease resistance, diagnostic and vaccine technologies, epidemiology, rumen microbiology and phytochemistry. In addition, systems analysis and impact assessment assist in targeting research.
In addition to such expansion of the relevance of ILRI's research, the Institute considers that it is equally important to base individual scientists and scientist teams outside Africa- Progress on this front was far less dynamic than anticipated due to slow donor response to this initiative. The Institute has made it clear that further expansion of these activities would only occur if entirely funded through project restricted resources. This implies ILRI's decision to preferentially treat its Africa - based activities. The Panel believes that this is a prudent and well justified policy, provided the Institute submits the entire Africa-based portfolio to continuous, rigorous scrutiny for quality, strategic value and output to confirm or reject its international public goods nature. If this was rigorously done, the opportunities lost outside Africa by applying the above policy could be valued correctly and put in perspective. As indicated in Chapter 3, the Panel has not found evidence that the Centre has the process and the procedures in place to systematically do that.
It is unavoidable under these circumstances that interpretation and implementation of the Centre's global mandate run the risk of being perceived as opportunistic. The Panel does not suggest that this is the case; it expects, however, that a compelling institutional vision and a concise institutional strategy, coupled with rigorous, systematic priority setting and effective and efficient planning, monitoring and evaluation of the Institute's portfolio - as suggested in Chapter 3 - will strengthen the Institute's global dimension very substantially.
The Panel believes that the Institute needs to promulgate a clear policy on its globalisation approach, which establishes criteria to be applied in both dimensions of this approach; in particular, this policy would have to provide the evidence for the strategic nature of the choice of location and system when ILRI platforms of capacity are established outside Africa.
In its collaboration with partners, ILRI is concerned that it may be hampered or affected by IPR concerns. This may be in regard to working relationships with NARS, ARIs or - perhaps even more so - the private sector. IPR concerns relate to genetic resources (e.g., plant genetic resources that ILRI holds in trust); to the property of others used in research (e.g., specialised proprietary materials needed in biotechnology research), or to processes or products that ILRI may discover in their own research (e.g., vaccines, methods). Matters of special concern include: access to and transfer of germplasm, IPR and patenting of the results of ILRI research; and use of biotechnology and proprietary materials.
ILRI holds one patent, 'Vaccines for the protection of animals against Theileria infection'; this patent has been granted in both Kenya (Pat. No. KE/P/90/00002, dated August 29, 1994) and the USA (Pat. No. 5,273,744, dated Dec. 28, 1993).
An ad hoc Joint Board-Management Committee developed a Policy on Intellectual Property Rights, Biosafety and Bioethics that was provisionally adopted for one year by the Board in September, 1998. After testing the draft policy for six months, a report and draft version for adoption will be presented at the September 1999 Board meeting. The Board also endorsed the establishment of an Intellectual Property Rights Committee to address issues of policy and guidelines, review and approve material transfer agreements, and to facilitate confidentiality agreements and proposals through e-mail.
The Panel welcomes development of an ILRI Intellectual Property policy and the plans for its implementation as the policy is tested in practice. The Panel understands that questions concerning IPR often require management on a case by case basis, and believes that ILRI is well on its way to resolving most of the matters that may arise.
The Panel believes that the IPR systems that are adopted by collaborating organizations need to be considered in relation to ILRI's own IPR policy and management system. In particular, it is important that IPR issues do not inhibit effective collaboration and that the requirements for entering into any agreement are considered rationally on a case by case basis. It would also be important to set up criteria for deciding when IPR agreements may be required, rather than having a blanket system in which such agreements are required for all activities.
The ILRI Policy discussed in 11.4 above also covers biosafety. The Board has authorised the appointment of a biosafety officer to oversee implementation of the policy. ILRI has a secure biosafety containment facility.
It is critical in the laboratory-based work that recognised standards and procedures are implemented, and that each laboratory has a clear-cut definition of its responsibilities related thereto. This is particularly important in relation to the pathogens themselves, genetic modification of pathogens and gene cloning where issues of possible escape into the environment may be involved.
As with biosafety, the ILRI Policy discussed in 11.4 also covers bioethics and emphasises four topics: equity; trusteeship of genetic resources; respect, responsibility and integrity in science; and social benefits.
ILRI's Animal Care and Use Committee meets on a regular basis and its guidelines are based on the very strict UK system.
The Panel commends ILRI for setting up effective bioethics regulations and for their systematic implementation.