INTRODUCTION
GUIDELINES
CONCLUSION
1. External Programme and Management Reviews (EPMRs) of CGIAR-supported Centres are carried out in accordance with the process Guidelines outlined below and the Terms of Reference (TORs) issued separately. Each review is expected to be strategic in orientation and as comprehensive as the situation warrants. To be credible and acceptable, all reviews must strive to be objective, transparent and participatory. The reports must be direct, explicit and frank. These principles are observed throughout the review process.
2. Being a member of a review panel is usually an interesting and rewarding experience. Moreover, Centre management and staff generally welcome the opportunity to discuss with panel members their achievements, concerns and future plans. A healthy atmosphere of mutual respect and collaboration in the interchange of ideas is the key to the success of the review. It helps to ensure that the recommendations of the panel are realistic, are well understood by the Centre management and staff, and will be willingly, or even enthusiastically, implemented.
3. EPMRs are expected to maintain high standards of quality and rigor, and be conducted by an independent and objective panel. The EPMR is expected to assess the Centre in terms of its mission and strategy, programme priorities, achievements and impact, quality and relevance of science, and effectiveness and efficiency of management, as noted in the TORs.
4. It is inevitable that the conduct of a review requires the collaboration of numerous individuals; as well as a process that enables the various participants to collaborate effectively in a complex assessment that has to meet high expectations and tight deadlines. The main participants in an EPMR are: the CGIAR Members; TAC and the CGIAR Secretariat; the EPMR panel Chair and members; members of the Centre's Board, management and staff; the panel's support team of external consultants and resource persons from the TAC and CGIAR Secretariats; and the Centre's many partners at the local, national, regional and international levels.
Roles and Responsibilities
5. The CGIAR, TAC and the Secretariats. The CGIAR establishes external review policies for the System, and EPMRs are conducted on its behalf, in accordance with the TORs and Guidelines endorsed by the Group. For each review, CGIAR Members are requested to propose Centre-specific issues for the panel to consider, and receive the review report. Once the timing of a particular EPMR has been determined, generally according to the 5-yearly schedule, TAC and the CGIAR Secretariat are responsible for the coordination and management of the EPMR, and they provide guidance on matters of review design and panel composition, in consultation with the Centre's Board and management.
6. Senior staff members of the TAC and CGIAR Secretariats serve as resource persons throughout the review process, accompanying the panel Chair and members during their visits to the Centre and on field visits. Besides substantive briefings on technical and management matters, they assist the panel on process matters, including the logistical aspects of report preparation and production. However, to help safeguard the EPMR panel's independence and objectivity, the Secretariats' resource persons are not normally expected to undertake substantive review, analysis or writing responsibilities on behalf of the panel.
7. The Panel Chair, Members and Consultants. The leadership and task management skills of the panel Chair are obviously critical, as are the expertise and experience of panel members. The Panel Chair is appointed by TAC in consultation with the CGIAR Secretariat and the Centre. The Chair's involvement begins early on, when he/she is consulted regarding panel composition, and briefed by TAC and the CGIAR Secretariat about the review process and key issues and concerns regarding the Centre. Once the review is underway, the Chair is responsible for ensuring that the panel undertakes its assessment and completes the task in accordance with the TORs and Guidelines for EPMRs. Given the magnitude of the task, the complexity of the issues, the fact that many panel members may by unfamiliar with the CGIAR, the importance of maintaining dialogue with the Centre, and the need to produce a report that reflects the consensus of the panel, the Chair's task is a demanding one.
8. Because the report should reflect the judgment of the whole panel, all members of the panel are expected to contribute to all aspects of the review report. Staff provided by the Secretariats assist the panel Chair and members throughout the process, as appropriate. Consultants are also provided to the panel, as needed, for limited periods of time, for assessment of specialized areas. While these consultants and resource persons from the Secretariats (and sometimes a TAC member) support the panel's efforts as members of a team, ultimately the panel is responsible for formulating the assessment and recommendations of the EPMR report.
9. The Centre Board, Management and Staff. The Centre's Board, management and staff play a crucial role in the conduct of the review. They are heavily involved in planning the review, and subsequently in organizing the review and preparing for the panel's visits to the Centre and to the field. Once the review is underway, it entails a significant degree of interaction between the EPMR panel and Centre staff, as part of a valuable two-way learning experience. Throughout the process, the collaboration and inputs of Centre management and staff are essential for the review to run smoothly and for the report to be credible and acceptable.
10. The Centre's Partners. Representatives of national agricultural research systems (NARS), regional fora, bilateral and multilateral agencies, NGOs and the private sector are important partners of CGIAR Centres, and their input is considered essential for the viability of the EPMR review process. As part of the review, representatives of such organizations are consulted for their views on the Centre's strategy, programmes and collaboration. This may be through panel visits and/or meetings, as well as through questionnaires or interviews. The panel may also visit or contact managers and researchers from other CGIAR Centres and other relevant institutions with which the Centre collaborates. Such consultations are valuable as a means of assessing the centre's role in the CGIAR and in the global context. Given the vast number of collaborators or potential partners of a Centre, such meetings must be limited. Their outcome is considered important, however, and is expected to feed into the panel's assessment of the Centre.
Panel Composition and Report
11. Panel Composition. The review panel is composed of experts in programme and management areas relevant to the Centre being reviewed who can carry out a comprehensive assessment and give the CGIAR their best judgment about the past performance and future potential of the Centre. The panel is expected to make an independent assessment based on its own observations and other information available to it, particularly the evidence provided through CCERs (see below).
12. The EPMR panel normally consists of about five members, including the Chair. Panel members are generally selected for their ability to focus on the institution-wide issues relating to the Centre's mission, strategy, priorities, programmes and management. To ensure adequate coverage of the TORs, the panel composition usually meets the following requirements: a) the Chair and at least two panel members are familiar with the CGIAR; b) at least two panel members have a technical background relevant to the Centre being reviewed; and c) at least one panel member has expertise in institutional governance and/or research organization and management.
13. Panel Report. The EPMR report is expected to present an accurate account of the outputs and impact of the Centre, along with convincing evidence attributing these outputs to the Centre. It is expected that in-depth reviews of particular programme or management areas would have been undertaken earlier through CCERs - and would not normally need to be undertaken by the EPMR panel. This enables the EPMR panel to concentrate on the important strategic issues rather than on specialized detailed assessments of each programme, project or activity.
14. Because research in the CGIAR System is a long-term undertaking, the problems the Centre is working on may not have visible outputs until several years from now. For this reason, the review report is expected to provide convincing evidence on the quality and relevance of the completed and ongoing research, and the efficiency with which the work is conducted, as a surrogate measure of the potential impact of the Centre's current programme of work.
15. Although the EPMR report is expected to be comprehensive, the panel has considerable leeway in deciding on what issues it would focus in depth. The review report highlights the most significant issues faced by the Centre and makes recommendations on how the Centre (or the CGIAR) could address them. It provides assurances and convincing evidence to indicate that other aspects of the Centre's programmes and management (i.e., those not covered by the panel's report in depth) are effective and efficient. It also comments on the effectiveness of the Centre's internal review system on which the EPMR was based, and on how well the Centre has addressed the recommendations of the other reviews commissioned by the Centre.
Integration with Centre Reviews
16. It is expected that detailed high-quality CCERs would have been completed within the 2 years immediately preceding the main phase of the EPMR. The CCERs are undertaken by specialized external consultants, assisted by members of the Centre Board and staff as resource persons (not participants). They are expected to cover the Centre's main research programmes (including their relevance and direction, achievements, science quality and impact) as well as key aspects of Centre management (including governance, research organization and management, finance and human resource management).
17. The CCER reports are expected to be made available, at the time of their completion, to TAC and the Secretariats. These reports, along with comments from TAC and Secretariat staff, are made available to the EPMR panel, along with reports of the follow-up actions planned or taken by the Centre's management and Board. Other analytical papers -particularly internal assessments of programme performance and impact - and other background documentation prepared by the Centre are also provided to the panel. The Centre is responsible for providing this information in an easily accessible and useable form, so that the EPMR panel's conclusions can be based on a comprehensive and thorough review of all aspects of the Centre.
18. The CCERs - which are often very detailed and comprehensive - provide essential evaluative information to the EPMR panel on particular aspects of the Centre's programme and management. Their availability in advance of the main phase of the EPMR helps create an integrated system of Centre- and CGIAR-commissioned reviews of each Centre, and enables the EPMR to be forward-looking and to focus more on strategic, rather than operational, issues.
19. The EPMR thus serves as a vehicle for analyzing, verifying, and synthesizing the information already available through CCERs and other reviews, and for making this information available to a wider audience outside the Centre. While the Centre's Board and management are responsible for ensuring that the internal evaluation system is sound (in terms of scope, coverage, quality and timeliness), judgments on the adequacy of a Centre's quality assurance system, including the processes for undertaking CCERs and other mechanisms of peer review, are the responsibility of the EPMR panel.
Board Assessment Visit
20. Interactions between the Centre Board and the panel form an essential component of every review, given the Board's important role in the CGIAR System. Hence, early in the process, prior to (or sometimes during) the first visit of the full panel to the Centre (see below), the panel Chair along with Secretariat staff and possibly one other panel member or consultant attend a Board meeting, and interview Trustees concerning Board and Centre matters. This helps ensure the participation of the Board in the planning and design of the upcoming review, including the identification of key issues and concerns of relevance to the EPMR.
21. The visit also provides the panel Chair and selected members or consultant an opportunity to review the documentation provided to the Board, interact informally with individual Board members, observe the formal meetings of the Board and its committees, and make an assessment of the Board's effectiveness and operations. This preliminary written assessment of the Board is made available to the panel (but not the Centre), and is modified as appropriate during the main phase of the EPMR (see below).
22. In assessing Board effectiveness and operations, the panel takes into account the key legal documents governing the Centre - particularly the Establishment Agreement, the Headquarters Agreement, and the Constitution of the Centre. It also keeps in mind the main provisions of the Guidelines for CGIAR Boards endorsed at MTM95, particularly the guideline on the "Role, Responsibilities and Accountability of Centre Boards of Trustees".
First Panel Visit and Briefing
23. Following the panel Chair's visit for discussions with Board members (or sometimes coinciding with it), the full panel undertakes its first visit to the Centre headquarters for about one week to ten days for initial discussions with Centre management, staff and Board members (when the visit coincides with the meeting of the Executive or Programme Committees of the Board). This initial phase visit enables the panel to obtain an overview of the Centre's current activities and future plans, to identify strategic issues to be covered by the review team, and to prepare preliminary drafts of key sections based on an agreed outline of the report which will be completed during the main phase several months later. During the first two days of this first visit, the panel receives detailed briefings from Secretariat staff and Centre management on the recent developments in the CGIAR and the Centre being reviewed, and on the processes, quality and content of the CCERs undertaken by the Centre.
24. Briefings by the TAC Secretariat resource person cover technical matters such as the CGIAR's mission, priorities, strategies, programmes and impact assessments; and by the CGIAR Secretariat resource person, management matters such as the CGIAR's governance, organization, finance and human resources. These briefings by staff of the Secretariats also cover the CGIAR's expectations regarding the scope and process of the review (as outlined in the TORs and Guidelines for EPMRs); as well as an overview of programme and management issues of relevance to the Centre being reviewed. The resource persons from the TAC and CGIAR Secretariats also provide substantive and process-oriented support as requested by the panel Chair. One of the resource persons, usually from the TAC Secretariat, serves as the panel Secretary.
25. The panel then receives briefings from Centre management and senior staff on the Centre's strategy, priorities, programmes, governance and management. These briefings focus particularly on the Centre's recent developments and achievements, CCER findings and conclusions, and future plans. In addition, the panel seeks additional information from other Centre staff, on a selective basis, as needed; and invites Centre staff members, either individually or in small groups, to voluntarily share their concerns, if any, regarding Centre-wide programme and management issues.
26. To help ensure that these briefings and discussions are as comprehensive and up-to-date as possible, and to enable the panel obtain a comprehensive overview of the Centre's work, the Centre is expected to provide to the Secretariats and panel members, in advance of the first visit, copies of the recent CCERs and other assessments undertaken, as well as other relevant Centre-related documentation (such as the latest Strategy document, Medium Term Plan, Programme and Funding Request, and other relevant policy documents or analytical papers prepared by the Centre). For the list of documents generally provided to the panel by the Secretariats and the Centre, see Attachment I. Most of these documents are off the shelf, and do not need to be especially prepared for the EPMR.
27. Towards the end of this first visit to the Centre, the panel spends two to three days preparing preliminary drafts and précis of sections based on the outline of the report agreed by the panel and the writing responsibilities assigned by the panel Chair. This ensures that the panel undertakes a significant amount of preliminary drafting prior to the main phase of the review, and continue its assessment of the key issues and concerns during the period between the initial phase and the main phase.
Field Visits
28. To help ensure that the EPMR panel's assessments are adequately grounded in the reality of the Centre's circumstances, the panel is expected to undertake country field visits, jointly determined by the Centre, panel Chair and the Secretariats. The field visits cover the major non-headquarters based operations of the Centre, so as to provide a realistic assessment of the Centre's field operations, working conditions, and interactions with NARS and others in the region. These visits by panel members (as smaller "sub-panels", if necessary) are often for about one week each, and are undertaken before the main phase of the review.
29. A senior staff member from the Centre normally accompanies the (sub)panel members on these field/country visits, but does not participate in substantive discussions with country officials or representatives of regional fora. The resource persons from the TAC and CGIAR Secretariats help coordinate the field visits and accompany the panel members, as requested by the panel Chair. These visits supplement any surveys of NARS and Centre staff, organized by the resource persons from the Secretariats in advance of the main phase.
Main Phase and Report Writing
30. The EPMR panel visits the Centre for a period of about three weeks to undertake the main phase of the review, and to complete and submit its report to the Centre's Board and management. As noted earlier, the Centre is expected to have made available to the panel, well in advance of this visit (through the panel Chair and Secretary), copies of CCER reports and other relevant documents; and the panel is expected to have completed the field visits and been adequately briefed by the resource persons from the Secretariats. The panel is thus expected to be reasonably well informed about the Centre and be familiar with others' detailed evaluations of its specific programmes and activities by the time it undertakes its own assessment of the Centre.
31. The EPMR panel's report is expected to focus on the four topics covered in the TORs - namely, the Centre's: a) mission, strategy and priorities; b) accomplishments and impact; c) quality and relevance of science; and d) effectiveness and efficiency of management. The report is expected to be succinct and written in plain language, focusing on strategic issues and proposing forward-looking recommendations on overall direction and priorities (rather than on detailed programme content or operational management). The writing style is expected to be direct, explicit and frank.
32. Since descriptive material and detailed analysis is expected to be kept to a minimum, a report of about 60-70 pages - with suitable cross-referencing (not summaries) of the CCERs -is expected. However, if the CCERs available to the panel are inadequate in quality, coverage or depth, the EPMR panel's report is expected to compensate for these gaps and weaknesses through its own analysis and assessment.
33. The drafting of the EPMR report is completed during the main phase, and the final draft chapters are shared with the Centre management to ensure their accuracy and completeness. The final EPMR report is normally expected to be completed at the Centre; and is presented by the panel Chair to the Centre Board, management and staff before the panel's departure upon completion of the main phase. Simultaneously, the panel Chair formally transmits the document to the TAC Chair and the CGIAR Executive Secretary.
Response and Follow-up
34. The Board and management of the Centre under review are expected to submit a formal written response to the EPMR report, addressed to TAC and the CGIAR Secretariat. Then TAC discusses the report in the presence of the panel Chair and representatives from the Centre (including the Board Chair and Director General), and prepares a commentary, in collaboration with the CGIAR Secretariat, including recommendations for follow-up action by the CGIAR or the Centre. The EPMR report, the Centre's written response, and the TAC and CGIAR Secretariat commentary(s) are then distributed to and considered by numerous individuals within CGIAR member agencies prior to its formal discussion by the Group.
35. As a final step in the review process, the CGIAR discusses the EPMR report and the accompanying comments at either its mid-year or annual meeting, and agrees on follow-up action. Occasionally, this follow-up includes a CGIAR-commissioned Mid-Term Review (MTR) to monitor closely the Centre's handling of major concerns raised during the EPMR. Generally, however, the Centre's progress in implementing the recommendations of the EPMR is followed-up by the next EPMR panel five years later, and are reported to the CGIAR - as a mandatory Annex of the EPMR report - thus completing the external review cycle.
36. EPMRs provide the CGIAR and other stakeholders very valuable information on the accomplishments and future prospects of each Centre funded by the Group. Because they undertake a comprehensive strategic assessment of all key aspects of the institution, a "clean bill of health" from an independent external panel can provide much needed assurance to the CGIAR Members - as well as to the Centre's Board, management, staff and partners - that the Centre is on the right track and has the institutional capacity to produce the desired research results. If significant changes in direction, scope, focus, or mode of work are needed, these too can be made on a systematic and periodic basis, based on Board-endorsed EPMR recommendations. This helps the Centre's programmes remain relevant, cost effective and of high quality, and the management efficient and responsive - as they should be in an international Centre of excellence.
