Why organizations need evaluation
Types of evaluation
Users of the evaluation results
Assessment process
References cited and background literature
Evaluation is an important tool for improving management. Through organizational assessment - commonly known as evaluation - the effectiveness of an organization is measured in terms of its functioning, problems and achievements from both the behaviourial and social system points of view (Lawler, Nadler and Cammann, 1980). Organizational assessment thus involves "measurement of variables related to patterns of organizational behaviour and effectiveness" (Nadler, Mackman and Lawdler, 1979). It can play an important role in helping managers improve the efficiency and effectiveness of their operations and can be an instrument for creating public support for the research programmes and outreach activities.
Depending upon when evaluation is conducted, it may be ex ante, current or ex post. Ex ante evaluation is conducted prior to implementation. It involves analysis of internal and external consistency of plans, programmes and projects before their implementation. Current or progress evaluation is conducted during implementation. It measures the extent of use of resources and materials, the execution of activities and the partial results reached in relation to the set plans. It is also known as monitoring or concurrent evaluation, since it takes place at many points during implementation. Ex post evaluation is conducted after implementation. It involves evaluation of the results and impact in relation to the set objectives.
Organizational evaluation "measures, compares and analyses the coherence between results and specific objectives and between specific objectives and general objectives of institutional projects, programmes or plans" (Hernan, 1987). It can be helpful in identifying:
· whether or not the objectives and goals originally established are being achieved, as well as their expected effects and impact;· whether the organization is adapting to new environments, changing technology and changes in other external variables so as to efficiently utilize the available resources;
· areas which need to be improved, modified or strengthened; and
· different modes to better fulfil the needs of the clients of the institute.
In addition, organizational assessment:
· generates evaluation information, which then becomes a valuable experience-based input in future planning, establishing of priorities and resource allocation;· furnishes financial data to justify the need for additional resources; and
· helps keep the key activities on the right track and offers information that allows the setting of minimum standards to promote compliance with the organizational research process objectives.
Depending upon the objectives of the evaluation exercise, assessment may focus on one or several of the following (Hernan, 1987):
· Economic impact This includes measurement of the effectiveness of research results, using techniques such as cost-benefit analysis.· Impact evaluation Impact is measured in relation to long-term effects on variables which were sought to be altered through the activities. For example, in a research organization, impact evaluation would measure effects of research outputs on transfer of technology and returns to the farmer.
· Basic evaluation This covers the identifying and analyzing of the socio-economic, biological, physical, technical and institutional aspects which can be improved by research activities.
· Analytical evaluation This involves socio-economic analysis of adoption studies, productivity analyses, risk assessment, use of labour, marketing credit and prices and their effects on technical alternatives.
· Operative evaluation This measures efficiency by comparative analysis between materials and resources used, activities carried out and the results achieved.
· Evaluation of results This includes quantitative as well as qualitative analysis of research results, retribution factors and probabilities of adoption.
· Traditional evaluation It involves use of reports, technical meetings, committees, ad hoc groups, courses and seminars.
· Personnel this covers evaluation of the performance of professional, administrative and technical human resources in the organization.
Sponsors are the primary users of the results of the organizational assessment exercise. Other users could include:
· policy and decision-makers who are responsible for instituting, continuing, discontinuing, expanding or curtailing programmes;· funding organizations which provide funds;
· target participants who take part in organizational activities directly or indirectly;
· programme management, by individuals or a group who coordinate during the evaluation programme;
· evaluators, who may be individuals or groups, and who design and conduct the assessment exercise; and
· organizations which compete for available resources.
Elements of assessment
There are typically three elements involved in an assessment exercise (Lawler, Nadler and Cammann, 1980):
· The organization, which is the main unit of the assessment exercise. Heads of departments, scientists and administrative staff of the research organizations are the sources of information.· The assessment team, which needs to use appropriate measurement tools to collect data regarding the organization and its activities.
· The people, who use the assessment results for making the organization more effective by setting priorities, policies, plans and research projects.
To assess the performance of any organization, one needs to carefully analyse the assessment situation with respect to some important dimensions, including those listed in Table 1 and considered further below.
Table 1 Steps in organization assessment
|
1. Objectives of the assessment exercise |
Objectives
The sponsors of the assessment exercise may have specific expectations: programme evaluation, scientific investigation, organizational change, etc. A clear understanding of the objectives and purpose of evaluation can provide the framework for assessment. For an agricultural research organization, the objectives might be to evaluate:
· the research achievements of the institute and its regional and sub-stations;· the objectives, scope and relevance of research programmes and budgetary support;
· whether research programmes are in keeping with the mission, objectives, policies and practices of the institute and in harmony with the programmes of related agricultural research institutes;
· the policies, strategies and procedures of the institute, and the effectiveness of working of different councils and committees and consultative groups responsible for micro-management within the institute;
· whether the institute's organizational structure is capable of providing functional working autonomy and delegation of authority in day-to-day working;
· any imbalance in the staffing pattern in relation to scientific, technical and administrative needs, and whether any changes in the set-up are required to make organizational functioning more effective and efficient;
· linkages established with the clients, farmers and other ultimate users of research results;
· budget allocations to the institute in relation to overall national plans, policies and long-term priorities;
· constraints which hinder achievement of institutional goals, and ways to reduce or overcome such constraints.
Size and nature of the organization
Although research organizations have a broadly similar mandate, they are also distinct from each other for a variety of reasons. Therefore one needs to understand the nature, type and size of the organization so as to be able to evolve an organizational model and then develop an appropriate methodology for the evaluation exercise. Some of the important dimensions to be carefully considered in this context are:
· demographic variables, such as age, size and type of the organization - factors which play a crucial role in determining the goals of the organization and therefore influence its performance;· type of activities of the organization. Agricultural research organizations are generally engaged in basic and adaptive research and so are concerned with the creation and development of knowledge and technology relevant to the agricultural sector. Evaluation of such organizations will have to be designed and conducted accordingly; and
· nature of the organization, based on details such as who are its clients, the nature of its output, who benefits from its outputs and services it provides.
Knowing the size and structure of the organization helps when designing an operational and economically feasible assessment methodology. Size not only includes the number of employees, but also its divisions and related institutions. In the case of agricultural research organizations, this covers various national-level institutions, their research stations at regional and other levels, and in many cases even agricultural universities.
Areas to be covered during the assessment exercise
Assessing the effectiveness of an organization requires money, time, human resources and advance planning. It also requires considerable contribution of time and effort from the staff of the organization, and hence diverts staff effort away from the organization's core activity of research. In order to justify and make the optimal use of this diverted effort, the methodology has to be holistic and effective. Therefore, designing a methodology is of great significance. It has to be specific to the type of evaluation exercise and generate information which is relevant, objective, systematic and comprehensive.
The evaluation exercise could be total or oriented towards a specific element, namely:
· general administration, which would aim at assessing whether the management structure for carrying out agricultural research is logical and in place;· the institute plan, which would focus on the institute's plan and examine its logic as well as fundamental components. It would also determine if the inputs needed (such as staff, services, facilities, land and funds) are adequate;
· staff, assessing their performance and their current capabilities in relation to the staffing requirements;
· support services, where the focus is on adequacy or otherwise of services required for conducting research. The type of support services required in agricultural research institutes will vary, but some common ones include a library and information service; laboratory equipment; service and maintenance facilities; administrative support; computer facilities; and land and growing houses for experimental uses;
· facilities and grounds to assess their adequacy for current and planned research work;
· financial issues, where the concern would be not only with current and projected flows of income, expenditure, cash and assets and liabilities, but also the marketing and promotional activities needed to generate funds; or
· tasks and organization of research, the organizational system and technical productivity.
Organizational goals are accomplished through various activities or tasks which are performed by groups or individuals. For performance of tasks, the organization requires individual skills and capabilities, an organizational structure, and rewards and sanctions. These have to be appropriately assessed in relation to the set objectives (Lawler, Nadler and Cammann, 1980).
Individuals are assessed on the basis of their skills, training, abilities, needs, desires, motivation, perceptions, attitudes and behaviour. These are usually assessed in the context of individual performance evaluation.
In a research organization, activities are assigned to groups of individuals. It is important to include in the assessment exercise variables such as the dynamics of groups, intergroup conflicts and the capabilities of groups.
Individuals and groups work within a formal organizational system to achieve common organizational goals. This necessitates a careful study of various elements of the formal organizational arrangements. This should include leadership practices, organizational structure and the mechanisms used for the coordination and control of individuals and groups. Informal organizational systems are a set of relationships, structures and processes that develop over time in the organization, generally as a result of individual and group behaviour.
An organization does not function in isolation from its outside environment. Its priorities, functioning, etc., are influenced by the external environment in which it exists.
It is, therefore, essential to consider environmental or situational variables which could have a significant effect on the performance of the organization. That would provide a real understanding of organizational behaviour.
Organizational performance can be improved by focusing on strategic aspects and better management.
Structural aspects include work flow, operations and information technology, departmentalization, hierarchy, control and information systems and de-centralization of decision making in the organization. These variables affect organizational performance in association with environmental and strategic variables.
Behaviourial aspects include leadership motivation and morale, cooperation and conflict, creativity, conformity and innovation in the organization.
Technical productivity measures the institute's outputs, results and impact. These are the raison d'être for the institute's existence. It is also perhaps the most difficult and controversial type of assessment area.
Organizational model on which to base the assessment
Organizational assessment covers a wide range of activities in and outside the organization. it is therefore essential for the evaluation team to identify the areas which should be covered in the assessment exercise. Organizational assessment models are helpful in identifying such areas. Some of the theories commonly used are:
· human relations;
· open system; and
· internal process and rational goal theory.
Choice of a theoretical base depends upon the assessment team's perception of organizational behaviour. The theoretical approach should be practical and comprehensive so as to make the evaluation exercise effective. The chosen model should explain the concept of the organization to be assessed, its structure and behaviour pattern likely to be observed in the organization. It should also help in identifying factors for observation and measurement during the assessment exercise. Some of these factors are:
· nature of the work;
· characteristics of the individuals;
· group functioning and nature of the group;
· the dimension of organizational structure and processes;
· environment outside the organization;
· characteristics of organizational effectiveness; and
· the relationship that exists among these factors.
Some criteria for assessing the effectiveness of an organizational model have been identified (Lawler, Nadler and Cammann, 1980):
· Explicit The model should be expressed clearly and fully, without which it may be difficult to measure effectiveness.· Theory The model should be based on scientific knowledge.
· Operationally defined The model should define the "constructs variables, effectiveness criteria and relationship among organizational factors" in operational terms. Only then can it give practical guidelines during evaluation and ensure a proper understanding of the model without any misinterpretation.
· Empirically validated The model should be tested to ensure that it represents the same relationship among variables as actually exists in the organizational setting.
· Face validity Since a model aims at communicating the results of the assessment, some face validity in it is necessary. The model should reflect the propositions and relationships which are in consonance with actual day-to-day behaviour of the organization members.
· Generalizability The model should be applicable to various settings, each composed of different variables.
Framework for conducting the assessment exercise
The organization is a complex social system consisting of different sub-systems. The subsystems in turn have various types of structures, processes and activities. Because of this complexity, it is essential to develop a comprehensive and integrated framework to assess organizational performance at four levels:
· overall organization level;· unit level;
· individual job level; and
· at the level of the relationships between jobs and units within the organization and also with other organizations.
The framework should able to assess different organizational units in relation to their total environment so that various units, structures, functions, etc., can be conceptually interrelated with each other. More specifically, the assessment framework should be capable of:
· identifying and distinguishing relevant characteristics of context, behaviour and performance at the macro-organization, unit and job level;· examining the unique design patterns of units and jobs that are differentiated vertically and horizontally within the organization; and
· determining how these units and jobs are integrated and the functional contribution they make to the overall performance of the organization (Andrew, in Lawler, Nadler and Cammann, 1980).
The dimensions to be examined include organizational context, design and performance at different levels (See EXHIBIT 3 in the Session guide).
Evaluation of data
This phase includes three steps:
· collection of information regarding organizational functioning and impact;· analysis and interpretation of collected information to gain an insight into the functioning, impact and effectiveness of the institution; and
· use of the information and analysis.
The objectives of the organizational assessment or the expressed needs of the sponsors and assessors themselves play an important role in deciding about assessment of data. They may have different needs and expectations from the assessment exercise. A knowledge of the members of the organization is equally important to identify the kinds of accessible data and the sources which are valid (see Table 2). The organizational model should help in specifying the types of variables and effectiveness criteria which need to be examined. Symptomatic data would provide guidelines for formulating hypotheses about the nature of critical organizational processes which need to be examined in depth. Symptomatic data are based on the collection of information and observations during informal data collection, which aims to identify the key issues, areas and behaviour relevant for the assessment exercise.
Table 2 Evaluating research organization assessment data
|
For a research organization, information should be collected in relation to different functions and fields, depending on the coverage expected in the assessment exercise. |
|||
|
Areas to be covered could include the following: |
|||
|
|
- Historical background |
||
|
- Organizational set-up |
|||
|
- Research programme and management |
|||
|
|
· identification and formulation of research projects |
||
|
· monitoring of research project |
|||
|
- Research evaluation |
|||
|
- Regional stations |
|||
|
- Research highlights |
|||
|
- Priority research areas for the immediate future |
|||
|
- Education |
|||
|
- Extension |
|||
|
|
· organization |
||
|
· research-extension linkage |
|||
|
- Administration |
|||
|
- Budget |
|||
|
|
· sources of revenue and income |
||
|
· expenditure by items to compute ratio of expenditure between actual revenue and salaries and administration costs |
|||
Obviously, these data will be generated through scientific and administrative staff as well as through the documents and records of the institute. Scientific information should be collected at three levels: institute director, division head and scientist.
Data collection methods
The organization is a social system with a wide range of complex activities and behaviour. To assess meaningfully the performance of an organization, one needs to collect comprehensive data on behaviourial, attitudinal, economic, functional and structural dimensions. There has to be an integrated approach, particularly when different data collection methods have to be used. That may be necessary sometimes to support, sometimes to supplement, and sometimes to crosscheck, data collected through other instruments.
Data can be collected through questionnaires, interviews, observation methods and focused group interviews (Lawler, Nadler and Cammann, 1980).
Interviewing is the most frequently used method for collecting a part of assessment data. People generally wish to talk about and share their problems, conflicts, perceptions and achievements. This basic human nature can be exploited through a skilful interviewer. If the interviewer is skilled, efficient and able to establish rapport with the interviewee, she or he can generate useful data.
Depending upon the degree of structure, the interview can be:
· unstructured;
· structured but open ended; or
· structured and closed ended.
These types of interviews provide question areas as well as alternative responses, such as liked, disliked, liked very much, disliked very much, etc. During the interview, questions may be accompanied by possible alternative responses, one of which can be chosen by the subject of the interview.
Different types of interviews generate different types of information. The choice of type of interview should be based upon the situation and the kind of information needed. While information can be obtained from group interviews in a time saving manner, there are matters which cannot be discussed in a group. Therefore the choice of type of interview should be based upon question areas and situation.
Questionnaires can be given to respondents or administered through structured interviews. When the questionnaire is self-administered, the subject chooses among the alternative answers and does the coding. The questionnaire method is economical, can be used on large populations simultaneously, and the responses can be easily tabulated and statistically analysed.
The questionnaire can have either open-ended or closed-ended questions, depending upon the goals of the. evaluation exercise, the nature of data required and the size of the population.
Observation is widely used for assessing organizational performance. It is planned, systematic and structured so that it provides required, meaningful and useful information. There should be a clear understanding of what to observe and what not; when to observe and when not; and what to record and what not.
Observations can be unstructured or structured. In general, unstructured observations are useful at the beginning of the assessment exercise as they can help identify the crucial variables and explain the nature and behaviour of the organization. Unstructured observation is participative. The observer acts as member of the organization and discreetly keeps observing events, and this can provide lot of useful information which other methods do not.
Observations can also be structured or semi-structured. In structured observations, the observer takes guidelines from the instruments and procedures and records the information. The choice of suitable observation should be decided according to the situational requirement. Observed data can supplement data collected from questionnaires or interview methods. They can also be used to crosscheck the validity of data collected by other methods.
Focused group interviews involve a group of eight to twelve or more persons assembled to discuss various aspects of a topic under the guidance of a group moderator. A trained and skilful moderator can extract highly useful information from the group through the technique of focused group interviews.
Focused group interviews are recorded using audio or video. Audio recording is transcribed later on. Video recording can provide an added advantage during the transcribing session. The transcriber can observe the gestures and facial expressions of the speaker and crosscheck previous conclusions. Transcription of the session helps in analysing the data and deriving some conclusions.
The focused group interview has the advantages of
· generating qualitative data as compared to quantitative techniques,· helping understand the behaviour of organizational members,
· providing useful data on the experiences of organizational members and their reaction to specific management problems, and
· being a good supplementary tool to enrich and validate data collected through other techniques. For these reasons, it is desirable to conduct some focused group interviews during the assessment exercise.
Documents and records During the assessment exercise, a considerable amount of data can be generated through records, documents and physical evidence. They can provide information regarding individuals, technology, groups and even the organization itself. There are many sources which can be explored to get this type of information (See EXHIBIT 5 in the Session guide).
The case study method is based on interviews with some of the organization's members about organizational background, culture, functioning, etc. The researcher writes a case on the organization, highlighting the relevant issues.
Field studies Here the researcher interviews members of the organization at different levels and collects data on some aspects.
Results, analysis and recommendation
The utility of the organizational assessment does not lie in the meticulous care with which it is conducted, but in the results and recommendations, which should be of use to the clients. Meaningful implementation of the recommendations is a critical phase in the organizational assessment exercise. The results will only be usable when they are in consonance with the needs of clients and are operational for improving organizational functioning (Edward, in Lawler, Nadler and Cammann, 1980).
Alderfer, C.P., & Brown, L.D. 1972. Questionnaire design in organizational research. Journal of Applied Psychology, 56: 456-460.
Caro, F.G. 1969. Approaches to evaluative research: a review. Human Organization, 28(2): 87-99.
Cummings, L.L., & Schwaf, D.P. 1975. Performance in Organizations: Determinants and Appraisals. Glenview: Scott, Foresman.
Douglas, D. 1986. Evaluation in national agricultural research. In the proceedings of a workshop held in Singapore, 7-9 July 1986.
Evenson, R.E., & Kislev, Y. 1975. Investment in agricultural research and extension: a survey of international data. Economic Development and Cultural Change, 23(3): 507-521.
Ferry, D.L. 1980. Measuring and Assessing Organizations. New York, NY: Wiley Interscience.
Hernan, C.G. 1986. Evaluation of agricultural research in Colombia. In the proceedings of a workshop held in Singapore, 7-9 July 1986.
Khandwalla, P.N. 1977. The Design of Organizations. New York, NY: Harcourt, Brace, Jovanovich.
Lawler, E.E., III, Nadler, D.A., & Cammann, C. (eds) 1980. Organizational Assessment. New York, NY: John Wiley.
Likert, R. 1958. Measuring organizational performance. Harvard Business Review, 36(2): 40-41.
Mirvis, P. 1980. Assessing physical evidence, documents, and records in organizations. In: Lawler, Nadler and Cammann, 1980, q.v.
Morse, E.V., & Gordon, G. 1975. Evaluation research. Annual Review ofSociology, 1: 339-361.
Nadler, D.A., Mackman, J.R., & Lawler, E.E., III. 1979. Managing Organizational Behaviour. Boston, MA: Little, Brown.
Norton, G.W., & Davis, J.S. 1981. Evaluating returns to agricultural research: a review. American Journal of Agricultural Economics, 63(4): 685-699.
Peter, H.R., & Freeman, H.E. 1989. Evaluation. London: Sage.
Price, J.L. 1971. The study of organizational effectiveness. Sociological Quarterly, 13(1): 3-15.
Santiago, F.M. 1987. Agricultural Research Evaluation in Latin America: A Literature Review. International Development Research Centre, Toronto.
Steers, R.M. 1975. Problems in the measurement of organizational effectiveness. Administrative Science Quarterly, 20(4): 546-558.
Van de Ven, A.H. 1976. A framework for organizational assessment. Academy of Management Review, 1(1): 64-78.
This training manual has been prepared as basic reference material to help national research trainers structure and conduct training courses on research management at the institute level. It is intended primarily for managers of agricultural research institutes in developing countries and for institutions of higher education interested in presenting in-service training courses on research management. The manual consists of ten modules, each addressing major management functions including motivation, leadership, direction, priority setting, communications and delegation. The four structural functions of management - planning, organization, monitoring and control, and evaluation - are covered in individual modules. The manual has been designed to support participatory learning through case-studies, group exercises and presentations by the participants.
