5.1 Efficiency, Efficacy and Expediency Tradeoffs
5.2 Scope of Review and the CGIAR Dependency on NARSs
5.3 Some Potentially Useful Instruments
5.4 Exploratory Innovations and Lessons at ICARDA 1993
One of the established traditions of the CGIAR System is the commitment to both planning and evaluation practices. The tradition of evaluation is the longer of the two. The rationale for both sets of practices is widely understood, is well-articulated (Özgediz 1993) and need not detain the reader here. There are concerns, however, that one or both of these sets of practices is rather "overdone" in the System. Extreme statements go so far as to argue that the System is being "reviewed to death." More sanguine observations are that the amount of resources consumed through review processes, and also planning processes as is observed in chapter 4, are consuming a significant share of a resource pie that is not growing and therefore, as must every other aspect of the System's work, come in for an ever-closer degree of scrutiny. Such thinking underlies the creation of a permanent Oversight Committee of the CGIAR being canvassed at the time of the Review.
It is patently obvious that efficiency aspects of the process must be - and indeed are being -examined critically with a view to achieving greater economies. TAC and the CGIAR Secretariat have already suggested many efficiency-oriented innovations and several (section 5.4) are embodied in the present Review. What is not so readily amenable to attention is the efficacy of the process. Most of the efficiency-oriented changes that have been introduced have been clearly economizing but they may have had some slight effects in reducing efficacy. Even this latter proposition is not immediately obvious as it may appear, for instance, that a small panel can reach consensus more quickly than one of the larger more traditional-sized review panels.
There are many unknowns in the issue of efficacy. That these remain unknowns is attributable to the inherent difficulty in, and also the additional cost of, "reviewing the reviews" and measuring any research-productivity-enhancing effects of reviews. Some attention has been given to this, however, although the findings are yet and perhaps necessarily somewhat subjective (Fuglie and Ruttan 1989).
Another enduring tradition of the CGIAR review processes is that of "objectivity" and "independence." Perhaps it is this tradition that has led to less consideration having apparently been given to the third of the "Big E"s of the title of this section, namely, expediency. It can be anticipated that, should the "budget crunch" continue, this aspect will necessarily rise closer to the top of the agenda of those who plan and manage the reviews of the System.
Several principles that might guide a more expedient approach include the following: (a) select panel membership with more emphasis on continuity rather than novelty of service, (b) experiment with extremely small rather than just smaller panels, (c) eschew field visits in favor of considering other forms of documentation concerning regional program activities, ideally supplied by both the respective center and a good cross-section of collaborating NARSs, (d) rely more and also more explicitly on input from those donors who already operate an effective and ongoing monitoring and evaluation system applied to investments in IARCs, such as USAID and World Bank to take not random but major-donor examples, and (e) use more resources from and more directly involve input from the personnel of the two Secretariats serving the System.
This particular Panel has not had the opportunity to reflect extensively and hardly even briefly on these complex questions. Because it is indeed a small Panel and the time allocated for the Review was already abbreviated, the opportunity for doing so was even more limited than it might have been in most cases in recent years. This Panel does, however, feel that the optimal solution in seeking the best point on the tradeoff frontier across these three "Big E" aspects has yet to be found. The Panel feels that it could have done a better job under different conditions - for example, (a) had it taken more time and especially had it devoted more effort to more detailed and lower level interaction in NARSs, (b) had it profited from deeper examination of programmatic issues with members of the research staff, and in doing so would have wished to have more disciplinary expertise than was available within the Panel, and (c) had it enjoyed quieter and more conducive working conditions at both Center and hotel.
In raising the issues of efficiency, efficacy and expediency here, the Panel does not do so with any sense of final or even interim judgement but rather to commend to TAC and the CGIAR Secretariat the likely value of close and probably continuing examination of where the best point in the tradeoff may be. It may well be that the situations of the now 18 centers are all so different that there will be 18 different optimal points and, should this be the case, there will be a rather greater requirement for fine-tuned planning for any particular review.
Art aside, perhaps, perfection is seldom part of the human experience. Human institutions are necessarily imperfect, despite whatever efforts are made to make them less so. Such is the natural situation in agricultural research institutions in general and, unsurprisingly, given the multiplicity of its designers and maintenance structures, surely is the situation in the CGIAR system.
ICARDA was established to work largely within the WANA region, with clear recognition of the difficulties, which range from complex and often troubled political situations between several nations, and even within some nation states, through to fragilities in both the natural environment in which agriculture is practiced and the institutional environment in which agricultural research is undertaken. Clearly, then, any reasonable assessment made of its work, either within its own organizational domain or with its partner national research systems, must be made against this background of difficulty and fragility.
Perhaps fortunately for the ER Panel, determination of the state of NARSs in the region is beyond its terms of reference. Some judgments about their state are, however, necessary in any endeavor to chart a course for ICARDA. Given the restricted opportunity for forming such judgments, the imperfection of the ER process itself must be emphasized, and acknowledged at the outset as a major qualification to the conclusions ultimately to be drawn. To the extent that these judgments prove to be harsh or insensitive to national aspirations, the ER Panel expresses its regrets for its bluntness occasioned by its search for insight to ICARDA's desirable path.
The NARSs of the region have developed unevenly, over space and time, from a stage (with a couple of notable exceptions) of rather general undevelopment around the time of establishment of ICARDA. The challenge faced by ICARDA, especially as recognition of the importance of institutional enhancement of NARSs has grown and gained declared importance and status among CGIAR objectives, has been and continues to be great indeed. Several aspects of the diagnosis of ICARDA's needs and research opportunities are best viewed as exercises in planning under uncertainty, although the extent of uncertainty has seldom been recognized as such in deliberations on these matters.
Indeed, on the basis of the "fieldwork" conducted in NARSs in the coverage of this ER, some tentative hypotheses can be advanced. In doing so the intention is to suggest some lines of inquiry that the Panel believes might fruitfully be followed up in a separate investigation, since it is clearly beyond the scope of any particular CGIAR ER and, indeed, beyond even the mandate of a center such as ISNAR. In fact, ISNAR "misses out" on at least two grounds, namely, its obligatory (resource-constrained) concentration on relatively small and but few national systems, and, perhaps more significantly in the current context, through its role as a collegiate "helper" rather than stern appraiser. The latter creates some fundamental asymmetries, the effects of which are shared to an extent by all CG institutions.
These asymmetries may appear somewhat trivial but since they are encountered in the context of all ERs, including especially the present one, it is perhaps worth belaboring the point, since other ERs have seemingly judged the topic as excessively thorny or too negatively political to raise. Some historical context may help introduce the topic in this brief consideration here. The "early generation" IARCs were widely regarded as useful, helpful, mission-oriented, sometimes successful, but often "arrogant," particularly in their tendency to ride roughshod over the sensitivities of sometimes fledgling NARSs, as they got on with the imperatives of making "two blades grow where but did one."
The varied successes and fortunes surely contributed to the growing recognition that, ultimately, it was NARSs themselves that would have to take much responsibility for generating the productivity gains in the less-developed world's food-producing sectors, whilst still protecting the integrity of the natural resource base. So it has come to pass that the CGIAR institutions now all feature an explicit commitment to help to build "capacity" (and, it is to be presumed, productivity) in NARSs. For an ER, this naturally leads to questions of NARSs and to the effectiveness of particular IARCs in fostering "capacity building," whether this be through formal "human resource development" initiatives or through various forms of research collaboration.
A major asymmetry is that NARSs, as NARSs, are seldom subject to the same type of ER process and, even if they were, perhaps the question of their overall efficiency might not come up, at least as it might be perceived by "outsiders" such as collaborating CGIAR centers. In fact, and not too surprisingly given the sensitivity of collaborative working arrangements, the centers have been remarkably reluctant to rate, or even be seen to rate, NARS research (or other) capacity. Thus, NARSs are being asked in some detail about the nature and quality of what they are receiving from the IARCs but the IARCs are obliged to be very circumspect about what it is that they believe they should be offering (differentiated by client), and are naturally inclined to overstate what it is that they have been and are offering (usually not very clearly differentiated by individual NARS recipient).
In keeping with the tradition of a focus on the reviewed Center, this ER too will largely be silent on assessing the NARSs with which ICARDA works, but other observers are not so confined. To take the case of just one NARS, and one judged by most as a relatively large, strong and mature WANA NARS, the World Bank (1992, p. 13) has opined that, in Turkey, the "main constraints for an effective development of agricultural research... have included:
(a) ineffective priority setting for planning, programming, budgeting, monitoring and evaluation stemming from inadequate policy guidance and... the coordination of all research in MARA;(b) the overall fragmentation of the former MAFRA's agricultural research with its 67 research institutes in four General Directorates and with partly overlapping responsibilities...;
(c) low staff morale due to inadequate schemes of service compared with other Turkish institutions and comparable positions (universities) and frequent transfer of research personnel...; and
(d) concentration of research activities, personnel and facilities in the more developed regions of Turkey to the neglect of research in the less developed regions..."
These and other difficulties are widespread in the WANA region and it is salutary to reflect upon them whilst contemplating ICARDA's achievements and challenges.
Readers may be surprised to encounter these discursive remarks in an ER Report. The Panel felt it useful, however, to place these concerns "on the table" to help develop the context, especially regarding NARS linkages. There are several prongs to the evolving CGIAR approach, notwithstanding its institutional expansion in the absence of resource growth, including ecoregional initiatives and NARS capacity-building emphases.
It is the latter that has particularly troubled the Panel as if has struggled to deal with the demanding questions that naturally arise, including the following questions extracted from the current Terms of Reference for ERs:
(5) Does the Center's strategy reflect a thorough understanding of the needs of the Center's principal clients and of the relevant activities of its partners and collaborators?(6) Are national authorities satisfied with the Center's strategy?
(12) How effectively does the Center's training program meet the needs of national research systems?
(43) How successful has the Center been in managing its relations with clients in developing countries...?
(44) Is the Center's strategy for collaboration with national research systems appropriate considering the sizes and stages of development of these systems? Are the priorities for collaborative work accorded to individual countries... appropriate?
(48) What contributions has the Center made to strengthening national research systems through training, institution building, collaborative research and technical assistance?
Since NARSs' "needs" and other related information are hardly assessable or even identifiable in a Panel visit of a day or two, the discomfort inevitably felt by Panel members will be all too evident. Worse, NARS leaders and other officials are practiced and skilled in sharing perspectives and situation analyses with members of the donor community, of the IARC community and visiting peripheral groups such as ER panels in a manner that is naturally (and properly) self-serving and that is not necessarily very readily incorporated into a balanced overall assessment of how and how well a center has been operating and how much better it might be.
These discomforting observations lead this Panel to suggest that some fresh approaches to ER tasks are required if NARS-oriented issues are to be adequately dealt with in future reviews. Minimally, it seems to the Panel, the same sort of judgmental torch might well, with their agreement, be shone on the relevant NARSs - perhaps on some rotational basis from ER to ER - as on the center itself. Only in this way could a holistic assessment of internationally or externally funded agricultural research interventions be properly made.
5.3.1 A Survey of NARS Clients and Collaborators
5.3.2 A Survey of ICARDA Research Staff
5.3.3 Other Approaches to Data Gathering and Analysis
The growing and proper recognition of (a) the importance of capacity-building work towards the sustainable functioning of NARSs and (b) the vital need to engage the research personnel of NARSs more effectively in the work of IARCs has been emphasized at several points in the Report (notably in section 3.3, sub-section 4.4.4 and more generally in section 5.2). It is thus consistent for the Panel to have sought the views of people in NARSs as a key input to this review.
This process had three main elements, two explicit and one implicit. The implicit element is that (unfortunately) only one of the Panel members happens to be from the WANA region NARSs. Indeed, he directs the largest NARS in the region and - quite apart from the disciplinary skills and research administration experience for which he was invited to join the Panel - brings a perception of and sensitivity to the NARSs' needs that otherwise would have been unavailable to the Panel's deliberations.
The explicit elements, were, first, the country visits made by Panel members, discussed elsewhere in this chapter, and second, the questionnaire survey of selected research workers and administrators in NARSs, administered by the TAC Secretariat. Description of the survey, the major results and some evaluative comment are all reported in Appendix 4. Meantime, a few comments on the process are in order.
Respondents are not chosen at random and hence the response data are not easily amenable to statistical analysis. It also clearly emerged in the qualitative responses of the respondents that individual bias, be it a local, institutional or personal bias, was strongly reflected in the responses. Individual bias and commitment to specific programs is inevitable in surveys such as this. At issue is the method of sampling and the eliciting of client opinion, which, in the view of the Panel, is essential to the Center's program. The Panel considers that the present survey method should be redesigned or, unless changed for the better, abandoned.
The Panel designed and administered a questionnaire of Center research staff, to solicit their views on a number of research management issues. The research staff responded in large numbers and many provided relatively thorough answers. In fact, some were very insightful and articulate. The responses were very useful in obtaining feedback on strengths and weaknesses of ICARDA's Management from the point of view of the researchers. It alerted the Panel to many issues to follow up on the second stage of the review. It was an efficient method (from the point of view of the Panel) for obtaining researchers' perceptions. It may not have been time-efficient from the point of view of the researchers, but it did give them the opportunity to provide input to the Panel on the management side in a way that can be synthesized relatively easily.
It should be pointed out that the results of the survey must be interpreted cautiously. The questions (Appendix 5) were open-ended so there was considerable variation in topics brought up in response. This means that it is difficult to assess how much support there was for particular ideas or attitudes. Another problem with this method is its tendency to surface negative comments more than positive comments. Both the analyst and readers must take this into account when interpreting the data. Nevertheless, this negativity is functional in alerting the Panel to issues that need to be examined further in face-to-face interviews with appropriate informants.
The Panel's experience with this method leads it to recommend its use in this type of review, but always with further follow up. It was pointed out to the Panel that the picture that emerges is bottom-up rather than top-down. It must be balanced with the Management view of the same issues. One suggestion for future use of this method is that a few close-ended questions should be included on topics for which the distribution of staff opinions is desirable. Nevertheless, the bulk of the questionnaire should be open-ended questions that give the respondents freedom to express their views in their own terms.
There are, of course, many other analytical devices and approaches that could find utility in a review of a center. For instance, a comparative analysis using financial data from several centers may help to throw light on the situation at one under review and this has been done in a number of recent reviews. Another process, adopted here, was to conduct interviews on an individual basis with all members of the Board, by a small subset of the Panel (including the Chair), and this seemed to work well. Other special-purpose analyses can be made, such as the one on researcher-turnover rate described in section 4.5.2.7, and these could be extended to individualized "skills analyses" if there were perceived to be disciplinary deficiencies in a center. Analyses of publication trends and citations may also provide useful input to a Panel's assessment.
5.4.1 The Combination of Program and Management Review
5.4.2 Enhancement of the Panel by Consultants
5.4.3 Active Involvement of Secretariat Staff
5.4.4 Shortened Main-Phase Visit
The main novel feature of the Panel's approach to gaining information from the Center was to attempt a transparent and iterative process. One aspect of this that has been used in some other recent reviews was the survey of research staff discussed in section 5.3.2; the Panel shared its summary analysis of the results of the survey with staff. Another possible departure from convention was the preparation of draft sections of the report, together with country-specific field-trip notes, that were shared with Center Management and staff in the weeks leading up to the second phase of the Review. This, in turn, permitted the Center to assemble written responses to some of the draft material for clarification of misunderstandings apparent in the draft documents. It is easy to "overdo" such a process of convergence of understanding but, on balance, the Panel was inclined to think that it was probably a worthwhile step. Some surprise was expressed by some Panel members about some of their draft material being circulated when it was not expressly written for such purpose. The Chair of the Panel regrets having circulated some of these unpolished documents, but observes that sometimes a succinct blunt note can serve to sharpen and speed an exchange of views. Notwithstanding this observation about dialogue with Center staff, the Chair also regrets that draft Review documents that had not yet even been reviewed by the Panel itself were made available to Board members arriving before the Review was completed.
The combination EPR-EMR is no longer a novel feature of reviews since it has now been done on several occasions and has been agreed to be the norm for the future. This Panel agrees with the advantage of such a combination and notes, in particular, that at least one issue, which should have been addressed earlier at ICARDA, rather "slipped through the slats" in the context of an Interim Management (only) Review. This is the matter of an "improved" central management structure per se having the claimed negative effects on research morale and efficiency. Combined reviews would, it is believed, in all likelihood avoid such potential oversights in the future. It does suggest, however, that even interim reviews should consider aspects of both program and management issues, even if they are concentrated on one side or the other.
One of the ideas experimented with in the ICARDA Review was to supplement a small panel by one or more consultants chosen to make for a more complete set of relevant disciplinary experiences and expertise. In the present case, two consultants joined the Panel as effectively full members for the first phase of the review, including some of the field visits. Without getting into particularities, the Panel feels that this arrangement worked especially well. The Panel found the contributions of the consultants to be very useful in their interim and main-phase deliberations. Indeed, taking this idea to its absurdity, one could contemplate a review consisting of just one person working with a set of specialized consultant reports.
A significant enhancement of the present "small panel" was achieved through having a more than conventional level of involvement and information-gathering activities by the two Secretariat staff members who participated in the Review. At the behest of the Chair, both of them for most of the Review served roles largely indistinguishable from those of Panel members. Both, however, chose not to be party to the Panel's decision-making process.
Donors doubtless wish to be assured that their expectation concerning the degree of independence and objectivity is met in the conduct of Panel deliberations. The Panel itself was conscious of this requirement but has no regrets to express about the active involvement of the two staff members of the Secretariats in this particular instance. Indeed, the Panel wishes to express its gratitude for their continued, constructive and energetic efforts towards the completion of this report on the tight schedule that was set.
The only real certainty in the main phase was the timing of the delivery of the final report to Management for reproduction for the consideration of the Board. Working back from this agreed date and deciding on a time that would be both more or less adequate for the task ahead but economizing on time and resources of both Panel members and the affected Center was, as it always surely is, a matter of some judgement and subjectivity. In this Review, the Panel - at the Chair's suggestion - decided during their initial visit to ICARDA to reduce the main phase from three weeks to two. In allowing merely a two-week period for this process, and with due regard to the iterative and consultative aspects noted above, the operation proved to be a very tight one indeed. There were surely several "costs" implicit in the tightness that emerged. One was the more restricted time available for consultation of individual Center staff members with the members and Chair of the Panel.
Another "cost," and surely a more serious one, was the highly constrained time available for the Panel as a whole to deliberate on drafts and to seek balance of judgement and consistency of treatment across material of disparate nature and refinement. The Panel sees this problem as yet another aspect of the imperfections alluded to in the opening paragraphs of section 5.2. That said, the Panel feels that it is sufficiently aware of the complex of issues raised by staff who approached the Panel and that more extensive consultation on such more personal perspectives and grievances may not have yielded much additional insight to its deliberations.
