5.1. Institutional design in policy research
5.2. Project selection and demand responsiveness
5.3. Project design and execution
5.4. Project communication and influence
5.5. Quality assessment
5.6. Personnel and recruitment policy
Chapter 2 of this report provides a general framework for assessing policy research. Figure 2.1 depicts the relationship between policy science, applied policy-making, and applied policy analysis. Chapter 3 discusses the multiple actors in the policy domain. Chapter 4 discusses changes in the demand for policy research. In this chapter, the supply of policy research is discussed and the quality dimension of this is analyzed.
Quality has many dimensions. One is the project selection dimension (discussed in section 5.1 below) where demand responsiveness is expressed. A second is project design (discussed in section 5.2). A third is project execution (discussed in section 5.3). A fourth is project communication (discussed in section 5.4). In each of these dimensions there are three quality criteria that should be met: (a) creativity, (b) competence in methods, and (c) interpretive insight.
Creativity is a quality that is difficult to define because the essence of creativity is "inventiveness." The "inventive step" in patent law is defined as the creation of something that is "unobvious" to those trained in the art. Creative policy research also has an inventive step. Competence in the use of analytic and empirical methods is more readily taught in graduate schools and acquired through experience and investment on the part of researchers in keeping up with the "frontier." Interpretive insight is closely related to creativity but has a more analytic and communicative dimension.
In section 5.5, we discuss some of the quality control or quality assessment procedures by which projects, programmes, and personnel may be rated and by which research policy managers and administrators can press for quality improvement in policy research. Finally, in section 5.6, we comment on personnel hiring and support policy as a means of quality control.
Public policy and public management research is subject to several sources of tension and potential conflict in CG Centres. These include:
a) Strategic versus applied research
b) Generic versus country-specific research
c) Single-country versus multi-country research programmes
d) Research versus advisory service and training
The conflict or trade-off in pursuing strategic (policy science, see below) versus applied research is related to the generic versus country-specific policy making and policy impact studies trade-off. Figure 2.1 provides a framework for discussing this issue. In this figure, a distinction is made between policy science, policy making research, and policy impact research. Policy science provides the analytic and empirical foundation for applied policy making and policy impact research. Policy science includes methods, both analytic and empirical, generic studies, and generic parameter estimates. (The term "generic" applies to methods as well as to studies that serve as guides or examples for other studies. Generic parameter estimates are useful in applied policy work where alternative situation-specific parameter estimates are not available.)
The CG Centres have a comparative advantage over all but the strongest NARSs in the production and dissemination of policy and management science. This comparative advantage (in fact this responsibility) stems from their relationship to the NARSs and their multiple client (NARSs) mandates. In terms of the CG system categories, they have a responsibility for strategic research, although it should be noted that the generic studies (parameters) components of policy science may be the product of good applied policy making or policy impact research.
The CG Centres have in general accepted a training and institution-building role, particularly where NARSs are not well equipped for advanced research. This can be a source of conflict with the policy science responsibilities of the Centres if it results in a sacrifice of methodological standards in the research. We understand and appreciate that the demand for policy research originates in country-specific situations and that timeliness is important. But, if CG policy research is directed toward meeting these demands at the expense of the policy science responsibilities, the CG Centres will have little more to offer than the consulting firm community.
We note this as a general principle. Our reading of the policy research of the CG Centres (particularly IFPRI) indicates that this research has generally been of a quality and character well above the consulting firm standard. We consider a substantial part of the work done in the Centres to meet the generic standard and thus to have contributed to policy science. We also consider the System's record on diffusion of policy science through collaboration with NARSs to be good.
We would note, however, that the CG Centres have not contributed as much to the methods component of policy science as might be desirable given their responsibilities to NARSs. While methodologists have been visitors and part-time staff, it is generally the case that CG staff are not methodologists (i.e., they may use methods, usually quite well, but do not produce methods). Applied university departments (agricultural economics departments) and strong policy research Centres in other fields (Brookings, HE) have generally invested more in methodological research and staffing than the CG system has.
We want to stress that our broader view of policy research, i.e., to encompass the production and dissemination of policy science, does not imply that policy science is not being produced by organizations outside the CGIAR (IARCs, NARSs) system. We would not expect the CGIAR system to be the major policy science producer for agricultural policy research. Many universities and other research organizations have contributed and will continue to contribute to policy science. However, the context to which this policy science is applied has specificities that correspond to the less developed economies and that eventually warrant adjustment in the theories and methods made available by policy scientists in the industrialized countries. IFPRI should consequently be aware that some investment in policy science may also be warranted because of this. The guiding rules in pursuing methodological innovations in policy research at IFPRI and at other centers where policy research is pursued are that they should normally be (1) by-products of solid empirical research and (2) developed in collaboration with external expertise.
The Panel consequently recommends that the forthcoming IFPRI review consider the issue of the desirable level of involvement of this Centre in policy science research and the approaches through which policy science research can be pursued.
There is also a conflict, or potential conflict, in the multi-country studies with a number of collaborators. The danger in such studies is that methodological rigor is sacrificed to accommodate collaborators. Again, we note this as a general principle. We also note below that the management and control demands for successful multi-country projects are high and that some loss of quality may have resulted.
Policy research in the CG system should be responsive to changing demands and to research opportunities afforded by policy science while meeting the comparative advantage responsibilities to produce and diffuse policy science (to NARSs). Chapter 4 of this report discusses the nature of changing demands stemming from economic and political changes. We might note here that there are two sources of demand that are created by the CG system itself. The first is the demand for research policy advice from system Centres. The second is the demand expansion associated with the addition of the natural resource Centres. These demands are partly responsible for the increase in social science research in the Centres in recent years.
The ability to respond to new demands is constrained by the inherent inflexibility of many social scientists regarding shifting to new fields of research. Social scientists specialize to achieve better mastery of methods and facts. Specialization is valuable, but it does produce some inflexibility. This has implications for personnel policy in the system (discussed below). There have probably been a few cases of "mismatch" between skills and problems in the system, but in general we consider the responsiveness in the system to be reasonably good. New hires and visitors have been utilized to achieve responsiveness.
There are special problems in responding to demands for research in a field of research that is itself poorly developed. The general demand for more work on environmental problems or on natural resource management presents a much more difficult challenge than in the case of food consumption and nutrition, for example. Data bases and methods often have to be developed. Choices must be made as to where to begin work. Again, we would note that the CG system does have responsiveness although there may be some sacrifices in quality in some cases. We should note that these project choices will have to be made in a system that is unlikely to be enjoying further expansion in the future.
There are two issues that further concern us regarding project selection and the possible conflict with the research quality responsibilities. The first of these is the growing proportion of special project funding and the impingement that this can bring to research projects. Good research units can handle a certain amount of contract research through creative fungibility. But when the proportion gets too high they become constrained in the same way that contract-based consulting firms are. It would not be a satisfactory situation if IFPRI were to become so constrained.
The second issue has to do with the growth of multi-Centre initiatives in the system. Many of these initiatives serve an important purpose. But they can be costly in terms of administrative and management costs. The CG system does not have sufficient policy research staff to respond to all demands placed on it at the present time. We are concerned that some initiatives may not only overburden CG resources, but end up creating networks and coordinating mechanisms with little to coordinate. This will require careful review after a few years of experience.
A final note regarding a traditional field of policy research where CG Centres have a comparative advantage. This is the field of multi-market computable general equilibrium models. We are aware of the high demands such models place on researchers an the need for revision and redesign. The CG system, however, should not be content with the view that F.A.O., the U.S.D.A., Iowa State, and others cover this field. The importance of having an in-house capacity for such modeling is highlighted in the recent IFPRI 2020 project.
In this section, we address two further issues. The first is the design and management of primary data collecting (including generic data collection). The second is the inherent problems with multicountry multi-PI projects.
We begin by noting that project design and execution are critical to the success of virtually all primary data based projects. The ICRISAT village studies illustrate the importance of both design and execution. They also illustrate the merit of generic design strategies. The older tradition in primary data collection held that it should be limited-purpose and designed to test well specified hypotheses. That tradition has guided many of the farm management type studies by system Centres. It is also part of that tradition, that field management and supervision was critical to the collection of data.
It is generally concluded that primary data from sub-Saharan African households is especially demanding of a proper statistical design and of field staff supervision and management. Many data sets (including some from CG Centres) from the region yield little insight because of inadequate quality control and constant checking and probing. The ICRISAT data sets were collected with a high level of field supervision and control. They were designed from the start to be generic data bases. And they have been so. The CG system should be producing more of these.
The matter of multi-country multi-PI research projects (with or without a primary data-component) raises additional problems of design and execution. This is because the skill levels of the collaborators vary and there is a tendency to lower standards to accommodate the project. For some projects, the multi-country value may more than offset any standard reduction effects. We are concerned that the expansion of initiatives may lead to overdoing some of these demanding projects.
On project execution, we would add a further point. Many of the policy studies use econometric techniques and tests and other advanced analytical methods. It is important that a system of review and advice be in place to support the appropriate use of frontier econometric methods and analytical techniques.
Policy research has an impact through several layers. Means of communication must exist for each layer. These layers can be roughly depicted as:
1. Policy science
2. Policy analysis
3. Policy advice
4. Policy formulation
5. Policy actions
Communication from Formulation (4) to Actions (5) is usually on administrative matters and typically does not require sophisticated economic modeling. Communication from Advice (3) to Formulation (4) may take the form of lobbying and formal and informal meetings (e.g., Congressional hearings). The advisor must translate technical economic language into non-technical language. Communication between Analysts (2) and Advisors (3) is more challenging (except when the analyst is also the advisor, as is often the case). This really requires formal means of communication. This form of communication may be technical (IFPRI Research Reports) or non-technical (IFPRI Policy Briefs). For some NARSs (where analysts advise on local policy matters), formal communication may not be vital, but for most NARSs and certainly for IARCs research must be published. This not only forces clearer communication but also enables a broader audience. For communication between Science and Analysis, formal technical communication in published journals and bodies is essential.
The culture of some of the IARCs (and other research Centres) has been to stress differences between their "real world" work and "academic" work. We accept this distinction as regards project selection (where academic institutions often are not very demand responsive) but we think it is carried too far as regards project completion and publication. Many weak NARSs justify the lack of serious efforts to publish and disseminate research results on the grounds that this is appropriate for academic institutions, but not necessarily for them.
It is generally the case that the weaker NARSs have not developed good systems of scientific communication. Financial incentives are often not provided for such activities. Meetings and memos replace the formal communications of science (and no one likes the pain of the critical review).
The IARCs in dealing with NARSs have to address this broad-scale problem. Research worth doing is research worth reporting, even when research has initially been in response to particular clients' demands. And, for the IARCs, this means reaching the broadest possible audience.
There are three (perhaps more) approaches to monitoring the quality of policy research. They are: 1) impact evidence, 2) user evidence, and 3) peer review (and administrative review).
1) For policy research, impact evidence is practically non-existent in a formal sense. It is difficult to associate policy improvements with policy research for several reasons. First, it may be difficult to identify actual improvements in a multi-effect setting. Second, many actors and interest groups impact on policy. Sorting out the effect of economic policy research from other influences is very difficult. It is our judgment, however, that the past decade (perhaps two decades) has been witness to significant policy improvements regarding agriculture in developing countries. These have taken place in trade and market interventions, in infrastructure policy, technology policy, and nutrition and health policy (and, even we would venture, in environmental policy). We cannot quantify the proportion of this impact associated with the CG Centres. The World Bank and other research units, partly because they have helped define a common set of policy recommendations, have contributed. Nor can one be confident that this policy research will be sufficient to prevent a reversal in these policy improvements. But there is good reason to think that policy research in the IARCs and NARSs has had an impact.
In the survey organized for this study. Centres were asked whether they have made any attempt at assessing the impact of their policy/management research. Of 13 respondents, nine answered negatively. Four answered positively, but indicated that they either had not yet obtained results or only had generated very limited information. The issue of impact assessment thus remains a fundamental and yet largely unresolved question.
The importance of documenting impact of contemporary and ongoing P&M work in the CG System has been increasingly recognized, notwithstanding the intrinsic difficulties in doing so in the face of impossible attribution problems in necessarily multi-player games. Work at IFPRI was launched in 1995 (with support from The Netherlands) to assess in a systematic and quantitative manner the impact of its work. The activity is coordinated by a Committee on Impact Assessment, and involves measurement of both direct and indirect impact on economic efficiency and fiscal costs, as well as subsidiary activities on the even more difficult to measure consequences for food security, nutrition, and poverty. This work has not been completed in any sense at the time of this Review, but it will be instructive to examine its progress in the context of the next IFPRI EPR, as the experience will have wide interest and relevance elsewhere in the CG System, and beyond.
The Panel recommends that Committee on Impact Assessment follow closely the IFPRI exercise in ex-post impact assessment, with the aim of learning from this experience how impact was achieved, how information was circulated, and how it was used in a timely fashion. Impact should preferably be assessed by experts from outside the respective Centres. Impact assessment should in particular be conducted before external reviews and should be accompanied by mechanisms for assuring internal feedback mechanisms and internalization of the information in a centre-wide learning process.
2) The user evidence for policy research is also very limited. Few, if any, user surveys have been made. One branch of user evidence, citations to IFPRI and other research in economic and policy publications, has not been systematically catalogued. This evidence is limited by the lack of good policy journals. The Social Science Citations Index offers this information for publications in scientific journals. Some Centre reviews (e.g., IMMI) have used user surveys but in a somewhat different context. Citations of CG papers should be more systematically analyzed.
3) Much of the quality control in policy research in the System must be achieved through review process of papers and reports. If a study is to produce a generic result, it not only must meet certain review standards but it almost always achieves improvement as a result. IFPRI is using reviews extensively and continued stress on quality, whether in a technical policy science publication or a more applied policy analysis, is strongly encouraged.
Striving for higher quality in research achievements needs to be a permanent concern of all researchers. In general CG policy research has been of higher quality than management research. However, overall, there seems to be an under-investment in mechanisms to assess quality of research and of outputs. Some of the research is peer reviewed through refereeing in scientific publications, but this applies to only a fraction of the research effort. External reviews are not peer reviews. Internally managed reviews should be used to seek guidelines for research quality improvement.
The Panel recommends that each Centre put into place effective mechanisms for peer review of-both their research programmes and projects (e.g., relevance, methodologies used, importance of the policy conclusions) and their research outputs (e.g., publications, public data bases, client satisfaction).
These recommendations for research quality assessment are not specific to social science research and apply to all CGIAR research equally forcefully.
Professional staff in all centres, and in particular IFPRI, ISNAR, CIFOR, and IIMI, should be well tooled in frontier developments in social science theory, management science theory, and research methods. This is fundamental for high quality strategic research, publication in refereed scientific journals, demonstration to policy analysts how to conduct policy and management research, and the teaching of policy and management research. This is also fundamental for IFPRI and ISNAR if they are to play a backstopping role for other centres in theory and methods. It implies that a sufficient contingent of in house scientific personnel must be at the theoretical and methodological frontiers of their fields. Centres engaged in P&M research should consequently compete on the job market for the very best applied theoreticians, be more aggressive in searching for candidates, and devote the necessary resources in recruitment. This also implies that Centres should seek collaborations upstream toward theory and methods, as seen fit, with universities and think tanks. Effectiveness of this collaboration, however, requires that Centres internalize in house enough frontier level scientific talents. Being explicitly watchful of maintaining this rigor is particularly important in a situation where dependency on soft funding is increasing, with the resulting pressures for immediacy in the delivery of results and risks of falling in the consulting trap in pursuit of funding. In particular. Centres must be careful to avoid permanent staffing on the basis of projects instead of long term programme commitments. As some scientific personnel moves upstream, a healthy balance needs to be maintained between practitioners and theoreticians, based on dialogue and complementarities.
Leading policy research Centres, particularly in universities, rely very heavily on recruitment and promotion policies to achieve quality. The weaker NARSs tend to use these mechanisms poorly, often because they are publicly funded and must follow civil service salary and promotion standards. They often do not provide attractive incentives (financial and non-financial) to recruit the best scientists (and they often recruit at a very early stage in the scientist's career, with the associated risks). They typically do not have high promotion hurdles. An important work of ISNAR has been to work on the design of management and compensation schemes that attempt to overcome these difficulties.
The IARCs do not have the luxury of imposing extremely high promotion hurdles to achieve quality because they have regional and local concerns. It is our assessment, however, that both the recruitment and promotion policies of the IARCs could be improved.
On recruitment, the IARCs tend to rely on a passive policy of position announcement to create an applicant pool. For some positions the passive applicant pools are relatively weak. There is room for more aggressive recruiting in the System, along the lines of the recruiting of economists by academic departments. For example, new Ph.D.s enter the job market at the AEA and AAEA meetings. The IARCs could be more aggressive in contacting them. In addition, intelligence on who might be available for short-term visiting assignments might be improved.
On promotions, we would also call for higher exigencies in the System. Promotion decisions tend to be somewhat ad hoc and might benefit from being more regularized. There are important benefits to a system that is even-handed, asks for high standards, and encourages young professionals.
Finally, the IARCs have generally used visiting scientists positions to achieve some new intellectual exchange. We encourage more of this, particularly in the policy science area.
The Panel recommends that Centres be particularly vigilant, and review their performance, in securing first rate scientific personnel at the theoretical and methodological frontiers of their fields, that collaborations with cutting edge scientists outside the system be increased, and that close cooperation be maintained internally between theorists, empiricists, and practitioners.
