The most significant change introduced to the external review process over the last two years has been increasing reliance of the CGIAR reviews on center-commissioned external reviews (CCERs). This change was motivated by three main considerations: (1) to encourage centers to rely more heavily on focused peer reviews for their own purposes, (2) to provide CGIAR reviews with more profound assessments of science and management conducted by specialists, and (3) to streamline the focus of CGIAR reviews.
From the outset it was recognized that centers could commission CCERs for a host of reasons, that some reviews may not meet the requirements of a CGIAR review, and that there might be instances when a center finds it inappropriate to distribute widely the results of a given center-commissioned review. The implication is that some center-commissioned reviews will not play a role in CGIAR reviews. These points are made in the CGIAR endorsed, October 1995 paper on CCERs (attached for reference.) Another critical point made in that paper is that CCERs are not substitutes for CGIAR evaluations, but can serve as building blocks for such evaluations if they meet certain standards. Moreover, to the extent that CCERs meet those standards, the CGIAR review can focus more on strategic issues.
The experience to date has been mixed. Some center commissioned reviews have permitted CGIAR reviews to reduce or eliminate their attention to the areas covered by the CCERs. Some have not been utilized by CGIAR review teams, usually because they were not intended to meet CGIAR needs or simply do not reflect the depth of inquiry required or because their results have not been formally considered by the center.
The CGIAR Secretariat and TAC remain convinced that CCERs have great promise and that the rationale outlined in the October 1995 paper remains valid. Even so, there is a need to join centers in working out a more effective modus operandi, e.g., defining what constitutes an adequate CCER for CGIAR review purposes (examples might be the way forward here), what should be the timing of CCERs (must they all be recent to qualify?), and to what extent can it be expected that all aspects of a center be accorded a CCER.
To the extent that CCERs are more heavily relied upon for CGIAR review purposes, those in the review process must be better apprised of their role and of their implications for the formation of CGIAR review teams. Clearly, the integration of CCERs in CGIAR evaluations will have implications for the role of review team chairs, for the size of review-teams, and for their make up. Given the experience to date, it is appropriate to initiate a concerted effort to review the logic of CCERs and, assuming a reaffirmation of their potential role, to redraw the lines for their effective utilization.
Two sets of information are needed to facilitate the shift towards greater reliance of CGIAR reviews on CCERs: (1) information on what constitutes a 'high value' CCER for the purposes of CGIAR reviews, and, (2) information on what constitutes a 'comprehensive set' of CCERs, again for the purposes of CGIAR reviews.
· The centers are free to commission whatever studies they need for their own purposes. However, to the extent a CCER is also to serve as the basis of the evaluation of that center by an EPMR, it should meet certain criteria and standards (such as externality, scientific standing of the people conducting the evaluation, whether the evaluation addresses quality and relevance issues, time spent on the evaluation, follow-up of the evaluation recommendations, etc.) We can outline the key features of what past EPMR panels have found to be 'high value' CCERs and, in cooperation with the centers, develop a set of criteria the centers could consider when commissioning their future CCERs.· Whether a center's CCERs are sufficiently comprehensive for the purposes of an upcoming EPMR could be determined by the chair of the panel conducting the review, with assistance from the Secretariat staff. This elevates the importance of appointing the panel chair early, so that he/she can review the available CCERs and give advice on the size and nature of the required review effort. Our impression is that CGIAR review panels could range in size from three to eight persons, depending on the availability, coverage, and quality of the CCERs.
In addition, the center's quality control mechanisms, including the schedule of planned CCERs, could be reviewed by an EPMR panel. The panel could assess the adequacy of the existing quality control mechanisms and make recommendations for improving them, including future CCERs that might facilitate the evaluation of the next CGIAR review panel. The adequacy of the CCERs for the future review would, of course, need to be assessed by the chair of the future review panel. In making this assessment, The chair of the future review panel would pay special attention to what the center has done in response to the recommendations of the previous panel on quality control mechanisms.
