4.1 Quality of Research, Service, and Training
4.2 Quality Management Process
The Panel has been asked to comment on the quality of ISNAR's research, service and training activities, in addition to its outputs and impact. Clearly, assessment of quality requires more in-depth study of ISNAR outputs and practices than is possible during an external review. Recognizing this, the CGIAR has encouraged the centers to initiate a process of "internally commissioned external reviews" (ICERs). Each ICER would bring together the disciplinary expertise needed, in adequate numbers, to make judgments about effectiveness and quality in a specific area of a center's work. The external review team would then have available a set of opinions of qualified experts that they can use to form their own judgments on quality.
ISNAR provided the Panel with eight papers as ICERs. One is the ICER on ISNAR's impact referred to in Chapter 2. Another is a think piece on restructuring, but not an evaluation of an ISNAR activity. The remaining six are evaluation reports of varying coverage on selected ISNAR projects or activities. These include studies on:
· ISNAR's training function;
· the Indicator Series;
· INFORM;
· IBS;
· the comprehensive institutional development program in Uganda;
· the PM&E strengthening project for Latin America and the Caribbean.
The Panel examined this set of six studies. It also reviewed the comments made by clients and observers of ISNAR about the quality of ISNAR's work, including the comments made by NARS and stakeholder representatives in the ICER on impact. Albeit subjectively, members of the Panel also examined some of the outputs of ISNAR from a quality perspective. Finally, the Panel examined ISNAR's quality management process. The comments made below reflect the opinion of the Panel on quality and the quality management process.
The ICER on impact notes that "NARS leaders rate ISNAR's overall impact on their organizations as positive and rate the quality of ISNAR's contributions as somewhat better than those of other institutions" (p. 13). It continues to note that "while the quality of many of these [advisory, research, and training] outputs is considered to be high, it is not uniformly so" (p. 14). The Panel concurs with these observations. Maintaining consistency in quality should be an important center goal.
In the research area, the Indicator Series project has generated several high-quality data outputs and a few highly regarded studies. The number of publications of ISNAR and its staff has increased over the years (from 695 for the 1986-90 period to 837 for 1991-96). The ISNAR publications list shows an average of 19 journal articles per year over 1991-95. This includes notes submitted to trade journals and articles which appeared in a special issue of Public Administration and Development edited by ISNAR staff in 1993.
Evidence on the quality of ISNAR's services comes mostly from comments of clients on the services provided. In some cases, such as IBS, there are comments on quality in the ICER. In this case the professionalism of ISNAR staff is applauded.
Client satisfaction with ISNAR's services varies by service and service provider. In the case of INFORM, for example, there is both evidence of high satisfaction as well as early reports of implementation difficulties - the latter stemming, in part, from the perceived inflexibility of the management information package. The Panel's interviews with stakeholders showed that there are some queries about the cost-effectiveness of the ISNAR-developed tools for resource-poor NAROs. Lowering costs would be desirable so long as quality can be maintained.
Comprehensive institutional development activities on which case studies or an ICER exist (such as Kenya, Morocco, Uganda, and Uruguay) illustrate that time spent in understanding the circumstances of the client (and involvement with a client over a long period) improve both the quality of the service as well as the implementability of the recommendations. This holds true particularly in cases where participatory approaches are used.
In the case of training, the Panel commends ISNAR for stressing professionalism in the conduct of training events by establishing a training unit. However, not all training projects make use of this available capacity. Also, it is surprising that participant evaluation data are available for only 28 percent of the 162 training events carried out over the last five years (as noted by the Panel that conducted the training function ICER).
A general impression of the Panel on the quality of ISNAR's work is that, as expected, excellence is linked with work by specific individuals. When one of these individuals leaves, the institution is left with a major void. The challenge for ISNAR is to widen excellence so that the center gains robust institutional competency in its priority areas. For this, ISNAR needs to recognize, reinforce, and elevate excellence throughout the organization on a regular and systematic basis. People with excellence should form the "intellectual pillars" of the organization. Such people often attract other people of excellence to join or be associated with the organization. Continuing links with other centers of excellence also strengthens institutional competency. With these means, the institution can extend the perimeters of excellence beyond the endeavors of a few individuals.
The Panel notes that quality can be enhanced best by establishing an institutewide "quality mindset". This means that the organization - from the Board to the individual staff member - regards quality management as a significant responsibility. The Panel recognizes that ISNAR has taken several steps over the review period to improve the quality of its work. However, more needs to be done to instill a "quality mindset" to the whole institution.
On the positive side, the Panel observes the following as notable quality-enhancing measures taken by ISNAR over the review period:
· Ensuring quality is explicitly recognized as part of the responsibility of the program directors.· ISNAR has initiated an anonymous peer review of all official center publications. This includes external reviewers identified by a centerwide committee. Publication requires prior approval by this committee.
· As noted above, ISNAR has initiated a system of ICERs. The Panel regards this as a potentially useful step towards establishing a centerwide "quality mindset". A number of additional observations are made on the ICERs below.
· The Panel considers the establishment of the training unit as a quality-enhancing action, as noted earlier.
· ISNAR has initiated a technical seminar series which facilitates peer feedback. Also, the annual internal program review serves, in part, as a forum for discussing quality concerns.
Areas requiring attention in enhancing quality include the following:
· As noted above, the Panel commends ISNAR for initiating internally commissioned external reviews. However, the ICERs should be exhaustive in terms of covering the major areas of ISNAR's work. ICERs on ISNAR's research activities (in the areas of both policy and management) conducted by internationally recognized experts on institution building and research management, for example, are a strong need. Similarly, periodic ICERs on key areas of service and internal administration and management would also be useful.In addition to their subject matter, the quality of the ICERs can also be improved. The ICER on impact commented on ISNAR's ICER mechanism and makes a similar recommendation. In particular, it is noted that each ICER should be conducted along the lines of clear terms of reference that spell out the foci and expected outputs of the review. In addition, the Panel suggests that future ICERs also have a focus on the quality of the work of the center, not just its effectiveness.
· ISNAR's performance assessment process can be improved to reinforce quality. ISNAR needs to demonstrate clearly to staff that "quality counts". When advancement is automatic - or is seen to be automatic - there is less institutional value attached to quality.
· Quality management should be an integral part of every activity at ISNAR. This means work plans should also spell out what would be done specifically to reinforce and monitor quality. The internal reviews should devote considerable time to discussion of quality. Likewise, the Board should take steps to ensure that the center has in place adequate quality management processes. The Board, in particular its Program Committee, should play an important role in the quality monitoring process.
· The Panel observes that when work is individualized (as compared with team-based), self-motivation rather than peer pressure acts as a driving force in quality enhancement. In such cases the institution needs to reinforce self-motivation through its incentive structure, the performance management process, and a strong learning culture. Client feedback is an essential component of such a culture and ISNAR has made several improvements in this area. Feedback from professional peers is equally important.
In conclusion, the Panel notes that ISNAR should bring quality management to the forefront of all its endeavors. While the steps it has taken in this area over the review period are significant, more needs to be done.
