A Discussion Note Prepared by
TAC and the CGIAR Secretariat
The Issue
Sources of Unevenness in the Quality of the Reviews
Rationale for a Proposed Approach
The Proposal
How This Proposal Would Improve the Quality of the Center Reviews
Operational Implications
Two Caveats
Next Steps
At the 1995 CGIAR Mid-Term Meeting held in Nairobi the Group asked TAC to study the question of consistency across external reviews and report at the International Centers Week on how it intends to deal with the issue. As responsibility for external program and management reviews rests jointly with TAC and the CGIAR Secretariat, this note was prepared by them and reflects their view on the future directions of the CGIAR's external reviews.
The note addresses the following questions:
· What is the quality or ''consistency" issue?
· What are the major sources of unevenness in the quality of the reviews?
· What is proposed?
· How would the proposed process improve the quality of the reviews?
· What are its operational implications?
· What steps should be taken next?
External reviews constitute a cornerstone of the CGIAR's system of accountability. To remain credible, they must maintain minimum standards of quality and rigor. Some recipients of recent reviews have observed that there is unevenness in their focus, scope, and analytical basis - thus, the request for this study.
While reviews are not expected to be carbon copies of one another in terms of their content, they are all expected to surpass a minimum quality standard. It is not acceptable to have even one review fall below an acceptable quality standard as this would threaten the credibility of the entire review system. Hence, the ''consistency" issue.
The term "consistency" is also used sometimes to refer to consistency over time, i.e., whether a panel's recommendations on a center contradict the messages given by the previous panel for the same center. This does not appear to be a major concern. In the survey conducted by the TAC Secretariat following the Nairobi meeting, the centers identified few inconsistencies of this sort. It is recognized that when there is some inconsistency, it is usually due to changes in the center and its environment (justifying a different recommendation) or is because the two panels genuinely differ in their interpretation or analysis of the situation. Therefore, while differences of opinion or judgment between two successive panels might occur, they should be pursued further to understand the underlying causes.
The real issue, and the one addressed here, therefore is how to modify the existing review processes so that all CGIAR reviews meet minimum standards of quality and rigor. This means that all reviews should present an accurate account of the outputs (and, to the extent possible, the impact) of the center, along with convincing evidence attributing these outputs to work by the center. Because research is a long-term undertaking, the problems the center is working on now may not have visible outputs until several years from now. For this reason, the reviews should also provide convincing evidence on the quality and relevance of the work in progress, and the efficiency with which the work is conducted, as a surrogate measure of the potential impact of the center's current program of work.
Several factors contribute to unevenness in reviews as they are now conducted:
a) Differences among review teams. Each panel is unique in its composition, leadership, and approach to the review. This introduces an inevitable element of difference in perspective, capacity, and rigor. The facilitators of the reviews promote evenness in the coverage of the questions listed in the common terms of reference and they backstop the panel in analytical tasks. But they, rightly, refrain from making judgments on behalf of the panel. Genuine differences remain, therefore, in the capacity and approach of the panels to conduct the task.b) Differences in available information. The panels reach their judgments based on their observations and review of Information put before them. Variations in availability of needed background studies (e.g., on the impact of the center or the quality of science as judged by peer reviewers) create vacuums, which, in most cases, are notably difficult to fill by the review team.
c) Insufficiency of yardsticks. Yardsticks are better defined in some areas than in others (e.g., germplasm improvement vs. institution building). Also, panels use their own judgment in choosing yardsticks that they feel are appropriate to the specific circumstances of the center. The Secretariats have made some progress in development of yardsticks for assessing center performance in specific areas, but more remains to be done. The review system must ensure that the yardsticks used in reviews accurately measure both the quality and relevance of the science that is practiced and the efficiency with which the center achieves its objectives.
d) Differences in coverage. Each panel is expected to carry out a comprehensive review and give the CGIAR its best judgment about the past performance and future potential of the center. The teams are charged to (a) assess the center's recent accomplishments and impact; (b) comment on its overall mission, strategy and priorities; (c) comment on the quality and relevance of the science that is practised, and (d) assess whether it is managed efficiently. Although each panel is expected to conduct a comprehensive review, it has considerable leeway in deciding on what issues the review should focus. As a result, some subjects are treated in greater depth by some panels than others, which gives an impression of unevenness across reviews. Again, this should not be a concern so long as the team addresses the terms of reference in a convincing way and provides credible evidence for not going into details in some areas.
e) Differences in review reports. What the reader sees is the final report of the panel, which carries the writing style of the chair and the panel. The tone and frankness of the reports also vary, as noted by several speakers at the Nairobi meeting. We advocate direct, explicit, frank, and short reports that address the terms of reference fully.
Last year the TAC Chair reported to the CGIAR on plans to redesign the CGIAR's review processes, which were endorsed at that meeting. The overall direction of the changes proposed last year still apply, but one can now go further than was originally proposed, given (1) the growth of center-commissioned external reviews (CCERs) throughout the system: and (2) factoring in the likely activities of the CGIAR Impact Assessment and Evaluation Group (IAEG). This permits sharpening the original proposal by focusing on one main review model, instead of having choices among several options.
Growth of center commissioned external reviews. The discussions over the last two years on the CGIAR's review processes have illustrated the advantages to the centers of external peer reviews of narrowly defined programs or subjects. Some of these include the following:
· CCERs help bring relevant specialized scientific talent in sufficient numbers to address a specific area of research (or management);· the shorter time periods involved in each such review make more reviewers available than are available for longer reviews:
· results are available to management in an even flow over time, rather than in one great lump - hence, are more easily considered and assimilated by the center;
· preparations for such reviews are less disruptive to the center's total activities than are large scale comprehensive reviews.
CCERs are still in early stages of implementation at the centers, and the centers are confronting several challenges in implementing their internal review systems, e.g., how to ensure that all significant activities are reviewed, that the review cycle is of an appropriate length, that the quality of the reviewers is of high standard, and that the observations of the reviewers are given appropriate attention by the management. We believe that these potential problems will be resolved - indeed there is evidence that some centers have done so. Even so, for the CGIAR's review process to rely extensively on CCERs, the Group must be assured that these reviews consistently meet appropriate quality standards.
Developments in impact assessment. The CGIAR has taken steps to strengthen die system's impact assessment activities as part of its renewal program endorsed at the Ministerial-Level Meeting in Lucerne. An independent Impact Assessment and Evaluation Group (IAEG) reporting to the CGIAR is in the process of being established, as is a "sounding board'' that would interact with the IAEG, made up of a cross section of users of impact information. Also, the centers are establishing an inter-center working group on impact assessment to learn from each other and help harmonize definitions, methods, and evaluation data.
As a result of these efforts, more comprehensive, accurate, and timely information is expected to be available on the impact of the CGIAR and its centers than at present. This will fill one of the major gaps in the CGIAR's current review process.
We recommend that the CGIAR move towards a single, integrated system for evaluating a center. This system would have three components: (1) a set of center commissioned reviews, (2) a mechanism for linking the center commissioned reviews with the CGIAR review, and (3) the CGIAR review itself. As the first and second components evolve and mature, the CGIAR review would have an ever sharper focus. We also recommend that the system move in this direction as quickly as circumstances permit. On our part, we are prepared to foster the circumstances that will encourage rapid movement to a single, integrated review process.
The proposed process and its underlying premises are more fully described below:
Underlying premises of the proposed process. The most important of these are as follows:
1. CGIAR-organized periodic external reviews are necessary to reinforce accountability, eliminate or minimize the need for separate reviews by each donor, and provide the partners and stakeholders of the CGIAR accurate information about the health and contributions of the center.2. Science and its quality can be evaluated best by a knowledgeable team of scientists, and the effectiveness of organization and management by experts in management.
3. External reviews of a center, whether commissioned by the center or the CGIAR, should complement each other as parts of an integrated review system.
4. CGIAR-organized reviews should continue to evaluate the center in terms of its:
(a) outputs (i.e., impact and research accomplishments); (b) priorities and strategies; (c) quality and relevance of the science that is practiced; and (d) management efficiency.
5. CGIAR reviews should continue to be conducted by independent panels of experts, where members serve in their personal capacity.
6. CGIAR reviews should produce short, frank reports, written in direct language, providing the panel's views on the most important issues facing the center, after examining thoroughly information pertaining to all four principal areas of the review (i.e., outputs, strategies, quality and relevance of science, and efficiency).
A vision of the future review process. A two stage process is envisioned. The first stage would involve generating a set of reviews on aspects of the center's work through peer reviews. This would be managed by the centers. The reviews generated during this stage would serve as inputs to the CGIAR reviews to be commissioned by TAC and the CGIAR Secretariat in the second stage.
A strong center coordinated review system would contribute both to good management at the center and to the success of the CGIAR's external review process. Center boards and management are interested in finding ways to enhance the quality and relevance of the science that is practiced at the center. There is no better mechanism for exploring this than specialist review teams made up peers. Such teams can address questions of quality and efficiency of science more thoroughly than the CGIAR panels which often have no more than one expert in each major program area. Also, center-commissioned reviews are more conducive to internal learning because they do not carry the "policing'' or oversight stigma often attached to the CGIAR reviews.
The center's internal review system would need to include studies or externally conducted reviews of the center's:
· impact;
· program achievements and science quality (usually in different program or disciplinary areas);
· organization and management, including the work of the board of trustees.
Each center's board and management should ensure that the center's internal evaluation system is sound in terms of scope, coverage, quality, and timeliness. However, judging the adequacy of a center's internal evaluation system should be the responsibility of actors external to the center.
The simplest way of linking the center's self-evaluation system with the CGIAR reviews would be by asking the center to forward a copy of the center-commissioned review reports to TAC and the CGIAR Secretariat on a regular basis (following their consideration by the management and board), along with an indication of the action being taken with respect to the recommendations of these reviews. This would enable TAC (in the program area) and the CGIAR Secretariat (in the management area) to study the internally commissioned reviews in terms of their adequacy for CGIAR purposes, e.g., that they meet appropriate quality standards. TAC and the CGIAR Secretariat would, then, be able to interact with the center about what, if anything, would be needed in the form of additional center-commissioned reviews before the CGIAR review commences.
This would imply that TAC and the CGIAR Secretariat would be interacting with each center about its plans and schedules for center commissioned reviews. If requested by the center, they would also provide assistance and advice on potential reviewers and on review methodologies. The Secretariats would be receiving a steady flow of center-commissioned reviews throughout the year. As a result, they would have more up-to-date information about the work of the centers than they now have, which would impact as well on the quality and timeliness of their advice and service to the CGIAR.
The CGIAR review would be conducted by a small (4-5 person) team of experts, supported, as necessary, by the staff of the two Secretariats. The review team would be expected to make an independent assessment of the four major areas of the review (impact, strategy, quality of science, and efficiency) based on the evidence provided through the center-commissioned reviews and their own observations. The CGIAR review would serve as a vehicle for analyzing, verifying, and synthesizing the information in the center-commissioned reviews. The review would highlight the most significant issues faced by the center and make recommendations on how the center (or the CGIAR) could address them. It would also comment on the effectiveness of the center's internal review system on which the CGIAR review was based and on how well the center has addressed the recommendations of the peer reviews.
Although the CGIAR review would most likely cover in detail only a few significant issues, it would provide assurances and convincing evidence to indicate that other aspects of the center's programs and management (i.e., those not covered by the issues addressed in depth) are effective. It would do this by reference to the results of the center commissioned reviews and its own observations.
First, the proposed review system would widen the coverage and improve the quality of the information base made available to the CGIAR reviews. Comments by TAC and the CGIAR Secretariat on review schedules, coverage and on potential reviewers would help the managements and boards to mount rigorous internal reviews that address the priority concerns of their individual center. One of the tasks of the CGIAR review panel would be to comment on the effectiveness of the center's internal review processes, based on their use of the reports of these reviews. This would also contribute to generating further improvements in the coverage and quality of center internal review systems.
Second, the proposed system will delineate more clearly the roles of the center-commissioned vs. CGIAR-commissioned reviews. The former will provide in-depth coverage of all program and management areas, whereas the latter will provide broad coverage of program and management questions, with a longer term horizon, and by taking the findings of the in-depth reviews as a base. Thus, the future CGIAR reviews would provide a more strategic assessment of the center's institution-wide effectiveness and evolution than the present reviews.
Third, in the long-run, the proposed system would foster greater transparency. Evaluation information about aspects of the centers' work would be made available on a routine basis with TAC and the CGIAR Secretariat. Thus, center-commissioned reviews would have an audience beyond the center. Also, feedback about the adequacy of these reviews would help improve their rigor, frankness, and quality.
Finally, more focused reports and summary sheets prepared in a uniform format (as illustrated in the review guidelines discussed in Nairobi) will foster greater consistency in appearance.
The centers would continue in the direction they recently started in commissioning external reviews, but this effort would need to be expanded and made more systematic, under the guidance of center boards and management.
TAC would be more engaged with studying the substance of the center commissioned reviews. This would be at the expense of review process questions which would be handled by the Secretariats. Similarly, the CGIAR Secretariat would examine more systematically the substance of the center-commissioned management reviews and assist centers in forming management review panels and provide suggestions on methodology.
From a process standpoint, a typical CGIAR review would proceed as follows:
1. TAC and the CGIAR Secretariat would be in communication with each center on an ongoing basis about plans and schedules for center-commissioned reviews, composition of review teams, review reports and the action taken by the center on recommendations.2. TAC would react to each of the reviews in a timely way, through its membership, as would the CGIAR Secretariat for the reviews on management. Then, about 18 months prior to the CGIAR external review, TAC/CGIAR Secretariat would indicate to the center their view on the overall sufficiency of center-commissioned reviews and what additional reviews, if any, would need to be completed prior to the CGIAR review.
3. TAC/CGIAR Secretariat would propose a rank-ordered list of panel chairs for the CGIAR review to the center, after reviewing the center's suggestions for chair. TAC/CGIAR Secretariat would appoint the chair following consultations with the center.
4. The panel chair, accompanied by essential resource person(s), would visit the center to establish dialogue with board, management, and staff and gain a firsthand impression of the issues and the available documentation from the center.
5. TAC/CGIAR Secretariat would appoint the rest of the panel, in dialogue with the panel chair and the center management. The members would be chosen on the basis of their skills in analyzing science-based organizations and the nature of the issues that need to be covered in the review.
6. The panel would hold its first panel meeting at the center or a more convenient location a few months before the main phase of the review. At this 3-4 day meeting the panel would be briefed about the review at hand and the panel chair would lead discussions about major issues, the adequacy of the documents at hand, and about the review process at the main phase. The panel could also consult with the chairs of the center-commissioned reviews, if it feels that this is necessary.
7. Following this, if the panel concludes that these are essential for addressing the issues identified during the first meeting, field visits would precede the main phase of the review. The main phase would last 2-3 weeks and would be held at the headquarters of the center. The panel chair would present the findings and recommendations to the board and management before the panel departs. The report is expected to be short (less than 50 pages) and the panel would complete it at the center.
8. Senior staff from the TAC Secretariat and the CGIAR Secretariat would support the program and management aspects of the review.
9. The Group's discussion of the reviews by TAC and the CGIAR is not expected to change dramatically. If there is need for follow-up by the CGIAR, this could be in the form of a mid-term review, as is the case now.
In terms of the review models (and terminology) that is currently in use, the model described here would replace the standard review format. The new model is a further extension and elaboration of the so-called "issue-driven review model" which was endorsed on a provisional basis at the Nairobi meeting. The "mid-term review" concept would remain as a possible review follow-up mechanism, as noted earlier. There would no longer be need for an ''interim review," because when all indications from the CCERs are that a center is doing well, the regular review itself would be conducted in a way similar to an interim review.
As noted in the beginning of this note, each CGIAR external review should satisfy stringent standards of analytical rigor and quality. Under the proposed integrated review model, this can be met only if all three components of the review process (the CCERs, the external assessment of their adequacy, and the CGIAR review) succeed in their tasks. Basing a narrowly focused CGIAR review on center-commissioned reviews alone would not be satisfactory. To the extent that the CCERs are judged unsatisfactory, the CGIAR review would be modified as necessary to compensate for their weaknesses and gaps.
Second, the transition to the process described above cannot be expected to be completed overnight. This would have a bearing, in particular, for the reviews that are scheduled to start in 1996 (i.e., ICRISAT, IFPRI, IPGRI, and ISNAR). Ideally, these should be conducted using the new review model. Postponing them will crowd the system's review schedule for 1997 and beyond. Most of these centers have commissioned some external reviews, but it is not clear if these are adequate to base the external CGIAR reviews on, in the manner described above. TAC and the CGIAR Secretariat would move immediately to assess those reviews.
If the CGIAR agrees to proceed along the lines described above, next steps would include the following:
· consulting with the centers and TAC to examine all aspects of the proposal, including its operational and cost implications, and refine the proposal accordingly;· modifying the terms of reference and guidelines for reviews (which were discussed in Nairobi) to reflect the new review model;
· interacting with the Impact Assessment and Evaluation Group, once it is established, to explore ways of integrating information on the impact of the centers into the review process (advice and suggestions from this group would also be invited on questions of evaluation methodology);
· working with each center to develop specific timetables and arrangements for self studies, including advice and assistance that could be provided by TAC and the CGIAR Secretariat in the commissioning of CCERs.
