PREPARED BY HUGH H. MILLER
PROFESSOR OF FORESTRY
UNIVERSITY OF ABERDEEN, UK
The purpose of evaluation and monitoring of forestry education and training programmes is firstly to ensure that they maintain their relevance. This is sometimes explained as relevance to the "real world", which begs the question of quite what "worlds" our graduates are being prepared for. The second objective is to review the standards of delivery, that is not so much what is being taught but how well it is being taught, in other words the quality of teaching. These two aspects of relevance of the teaching and the quality of the teaching need to be kept separate.
In considering the matter of relevance, the issue is sometimes considered to be simply a matter of determining the desires of current employers. This is certainly important but can be dangerously restrictive. Firstly, the matter of defining these employers may not be straightforward. There will be a range, often a fairly small range, of traditional employers, usually headed up by the local state forest service. They undoubtedly have to play a major role in any evaluation process for they are dealing with the problems of forest policy implementation and forest management at first hand. However, even the most radical thinking of them will bring with them the conditioning of their own training and the ethos of their service (often a powerful factor among professional foresters). They are perhaps not the best to articulate the needs of future employers. Neither can they speak for potential employers beyond the normal confines of forestry, because it must be kept in mind that our discipline is currently rapidly widening in scope. If this greater breadth is not acknowledged the future for forestry as a separate subject may be very short.
The evaluation process, therefore, should be open to interests well beyond the narrow confines of what was once conventional forestry. Professionals from other land-based disciplines are obvious candidates and the more critical they are of forestry the more the need to include them. Criticism is never stifled by exclusion. Criticism is without reason, even if it may be poorly informed.
The issue of teaching quality is easier dealt with, for many people know, or believe they know, what is good teaching. Indeed teaching quality can be very effectively monitored and evaluated within an institution by individuals with no background in forestry at all but who have experience and knowledge in teaching. Such experience and knowledge is not the only requirement for an important input must come from the students themselves, for which there must be an appropriate procedure to ensure that the students feel able to make an honest and forthright contribution.
The process of monitoring and evaluation must necessarily differ between education at university degree level and training at technical and worker levels, which in turn will differ between them selves. It is among university graduates that the innovators and advocates of future change should be found. The education they receive must encourage critical thought, must foster the ability to dream up other worlds and even promote risk taking. These characteristics are every bit as important as technical competence. It has to be admitted that these graduates are seldom much opportunity to institute change until they are well into their careers which means they have to remain critical and maintain flexibility throughout their working lives. The means of imparting of such skills within a degree programme is not easily defined and accordingly far from easy to evaluate or monitor.
The technical forester, who seeks a career planning and supervising forestry operations, needs rather a different set of skills to the innovator discussed above. Imagination, while important, is less significant than skills in organisation and person management. If such individuals are largely produced by educational organisations other than universities they will go through a purpose designed training programme that should be evaluated accordingly. In this case emphasis in evaluation would be expected to be given more to demonstrated practical competence. Increasingly, however, this cadre of the work force is being recruited from among university graduates, which makes for problems when designing and evaluating curricula that now have to produce two rather different individuals (a problem by no means limited to forestry).
For forest workers training is usually given, if at all, in short very specific courses usually supplemented by appropriate on the job training. Assessment would be in terms of competence to carry out specific tasks. Evaluation would have to relate to the relevance of such tasks to the calls employers make upon their workers, not least in relation health and safety either as defined by the employing organisation itself or, increasingly, by the state.
As suggested above, evaluation is about maintaining and improving relevance of the teaching and the quality of deliverance. The relevance of a curriculum can rapidly date. The demands placed on the forestry profession and its work force has changed in certain profound ways over the past two or three decades and there is no reason to believe that this rate of change is about to slow. The nature of such change is the subject of other papers at this meeting but because of these any students still being educated or trained on a curriculum of the 1970s would be somewhat puzzled, and singularly ill-equipped, when first entering the work force.
It has already been suggested that one purpose of education, particularly at university level, to produce individuals that are willing to embrace change and preferably have the ambition to lead it. Such change, however, should not be change for change sake but rather a response to changes in demands placed on the profession, particularly the demands of society at large and of specifically knowledgeable groups within society (e.g. NGOs). Education, therefore, needs to instil not only a capacity for rational criticism but also, and most importantly, the humility to value and respond to the views of others. To achieve this a degree programme should be able to demonstrate that students are exposed to a wide range of views and that, outside the immediate technical areas, they are not required to believe that there can only be one right answer. At all times they should be encouraged to debate issues among themselves and with the teaching staff. Therefore, modern curricula should be expected to include task-based training, student debates and presentations.
When discussing what to evaluate consideration has to be given to both content and balance. By content is meant the subjects covered in the different courses making up the degree programme and by balance is meant both the balance of subjects in the individual courses and the balance between courses. These should be monitored continuously and periodic evaluations carried out to determine and instigate change. Inevitably both content and balance will to some extent reflect the abilities and interests of the available teaching staff. The evaluation process should guard against this. Change, however, can only be effective if the relevant staff is sufficiently committed to willingly instigate the change in the courses they teach.
An evaluation can be instigated or carried out at various levels. No evaluation would be effective if it did not involve the teaching staff of the faculty/department. They alone, however, are generally insufficient because of both institutional inertia (the tendency to resist change) and the fact that academics, even in the best organised institution, are unlikely to be fully aware of all the changes besetting the profession at large. Advice is required from outside the faculty and, most importantly, from outside academia. Such outside influences should be involved both in any periodic review of the curriculum and in monitoring of performance so that change can be instituted timeously.
What kind of interests should be included in such evaluation and monitoring processes? As well as those professionally involved in forestry there should be representatives of allied professions. The most obvious of these are agriculture, ecology and conservation, fisheries and wildlife, rural development, and tourism or recreation. Sometimes one individual may be able to represent two or more areas of interest, such as agriculture plus rural development or fisheries and wildlife. The areas of interest may be represented by government officials, private practitioners (e.g. farmers), or representative from appropriate NGOs. Drawing some representatives from the latter can bring important new insights and win friends (in some countries the NGOs are becoming increasingly important employers of forestry graduates).
Of course, as well as such outside interests any evaluation panel or advisory board should include representatives of both the forestry profession, perhaps including arboriculture, and the timber processors. Forestry is becoming a very broad profession and it can be difficult to balance the various interests. It would seem rational to select representatives of the main organisations employing the forestry graduates from the university or college in question. However, in so doing an attempt should be made to look ahead at the likely future balance of employers, rather than simply reflecting past trends. Consideration should also be given to ensuring balance between main stream forestry employers and representatives of the allied professions discussed above. The latter should not be left to feel that the are hopelessly outnumbered on the board.
Historically, and still currently in some countries, one organisation, usually the state forest service, has essentially dictated what qualities a forestry graduate should possess. If this is the case such an organisation should be well represented on any evaluation panel. However, because of the somewhat conservative nature of state forestry services there may be problems if their representatives are given too big a role to play. In such circumstances selection of individuals should be made with care. In many countries the day of one dominant employer seems to be passing, a result in part of privatisation, in part of reductions in numbers of civil servants, and in part of the widening role that foresters are being asked to play. This has left a vacuum that can be filled in a variety of ways. One possibility is that the setting of requirements for a forestry education be left to the universities or colleges alone. This brings with it the very real danger that the provision for the teaching of forestry-specific subjects in the curriculum may be reduced in the interest of "efficiency" or to cope with the staffing problems that seem to be faced by so many educational institutions. Clearly, in such an event evaluation and monitoring by outside interests can have an important role to play in preventing forestry education or training being devalued.
In some countries concern about the professionalism of foresters has lead to the establishment of professional bodies with the responsibility to define the education standards for entry to the profession and to monitor performance thereafter. Thus the appearance of individuals variously titled "registered professional foresters" or "chartered foresters"; somewhat analogous to the position in professions such as accountancy, law or even medicine. In the absence of a dominant employer, and where there is increasing employment in the private sector, there is much to be commended in this. Indeed, the quality of the profession may depend on such developments. Where such a body exists clearly it should be represented in any evaluation or monitoring process for the qualifications being reviewed will have to accord with the stated requirements of such a professional body.
Experience with a Departmental Advisory Board at Aberdeen suggests that that participants should be invited for their own specific expertise and not as representatives of particular bodies, such as a specific NGO or government department. If such bodies are merely asked to send representatives what seems to happen is that with time enthusiasm may wane and the task can be passed to a fairly junior employee, sometimes selected for his/her nearness to the university rather than relevant experience and seniority. Generally, therefore, the invitation should be extended to named individuals who occupy positions of influence in their organisations. A potential problem with this idea of inviting particular individuals is that they may remain on the panel long after they are in a position of any significance. Accordingly, appointment should usually be for a specific period, usually three years but renewable for at least one further term. Perhaps there is an exception to the rule of only inviting individuals when considering organisations such as professional bodies where the senior members only hold elected office for one or two years. In this case the invitation should be to specific office bearers, such as the President or Chairman of the Education Committee.
Consideration also has to be given to the role of the students in both evaluation and monitoring. Their contribution to the definition of what is what should be included in a forestry curriculum is necessary limited, but nevertheless they should be given the chance to have their say. In this regard canvassing the views of graduates of three of four years previously can be very instructive. Current students have a large role to play in monitoring the delivery of teaching, not only the quality of teaching but also in identifying areas of overlap.
It is difficult to lay down specific criteria and indicators for the use in the evaluation of the relevance of forestry curricula or training schedules. Relevance has to be judged against the requirements of a range of employing organisations, organisations that cannot be expected to in the first instance to agree among themselves as to what is required. Evaluation needs to be an iterative process during which the various players come to understand the needs of others and to agree compromise. This is further support for the idea that members of an evaluation panel or advisory board should be appointed as individuals and not as mandated representatives of their organisations. In the latter case it will be hard, even impossible, to reach agreement on some issues.
In an evaluation the first process should be to reach agreement among the players as to the objectives of the education or training programme. Only once this has been achieved is it possible to review whether these are being achieved by the existing programme and so recommend necessary change. The criteria and indicators, therefore, will emerge from an initial process of dialogue and consultation and necessarily will vary somewhat between countries and between institutions within a country.
Where a major evaluation of a curriculum is to be carried out considerable planning is required and the process might be quite extended. The steps may be something like this:
This is a major process and not always approved by institutions jealous of their academic freedom. Compromise may be necessary but it should be understood that this is a compromise.
Step 7 above implies that the evaluation panel remains in existence after the initial task of reviewing the curriculum. This may be the original panel, a reduced version of it or an entirely different group. At all events experience in the University of Aberdeen has shown that a standing Advisory Board for forestry education is extremely useful. Clearly the input the members of such a board can make is potentially invaluable in maintaining the relevance of a curriculum. Their role is one of ensuring continuous review and gradual improvement of sections of the syllabus. In the normal course of events they are unlikely to be in a position determine the details of a major revision hence there remains the need to periodically carry out a root and branch evaluation along the lines suggested above. However, the Board is in a position to identify when there might be a need for such an evaluation and consequent revision of the curriculum. In practice this decision is likely to be reached jointly by the faculty staff and the Board.
For an Advisory Board to function effectively it is important that the Dean and Department heads are willing to bring all appropriate issues to the attention of the board and that the views of the board are welcomed. Furthermore, the board must be kept fully informed of the actions taken as a result of their recommendations. If people of adequate seniority are to remain on the board they must be assured that their time and expertise is being properly used.
From experience one particular advantage of having to report to an Advisory Board is that at least once a year the faculty has to review progress and question the status quo. Without this spur it is very easy to become complacent over the curriculum.
The next point to be considered is the monitoring of the quality of the teaching standards. All educational establishments should aim to develop within their bureaucracy mechanisms for ensuring teaching quality in all subjects and at all levels. Checks should be put into the system to ensure that the courses making up an education or training programme are properly planned and that what is planned actually happens. Evaluation of student performance (continuous assessment or exams) is one indicator of success in this regard. Ideally the standards being achieved should be periodically checked by someone from outside the university or college to ascertain that they match the standards being achieved elsewhere. For some subjects, such as medicine and engineering, such quality control may be carried out by the appropriate professional institute. In forestry this has seldom gone further than a mechanism for the approval of courses or programmes as being suitable for entry into the profession. It is also preferable that the system established within the bureaucracy of an institution to monitor teaching quality should be subject to periodic audit from outside.
A major aspect of teaching quality audit is ensuring that the views of the students are listened to. There views can be obtained through questionnaires (the issuing of which by course co-ordinators should be compulsory), through student-staff liaison committees and through the normal pattern of dialogue between students and staff (often at its best on field trips). Questionnaires are important, they should be standardised across the institution, the summarised results be seen by the Dean or Department head whose comments should be sent with the summary to the committee (teaching committee) responsible for the quality audit within the institution. Any teacher who this process reveals to be a consistently poor performer should be investigated and if necessary encouraged to undertake further relevant training. The kind of questions that should be put to students (to be marked on a four or five point scale) could include:
In the form used at Aberdeen there are 50 such questions (arguably too many).
There are two questions here. Dealing first with instigating an evaluation system. Although the sort of Advisory Board discussed above has the prime responsibility of evaluating in the sense of continuous rather minor change, the appointment of such a board should probably be the first step in instigating any evaluation system. Deciding whether or not to instigate a major review of the curriculum should be something that arises from the discussions in this board. The procedure that then might be followed for such a major review has already been discussed. Appointment of the Board and the decision, if at all, to involve it in a curriculum review must be approved at the highest levels within the university/college concerned. Without such support progress might be difficult.
An Advisory Board once appointed should meet at least once a year, with the option of further meetings if called for. That this Board should meet regularly at about the same time of year is important, otherwise its members will cease to treat it seriously.
The second question relates to the monitoring of teaching quality. Ideally the mechanism for dealing with this should be established at the highest level in the university or college concerned and the resulting procedures applied equally across all faculties and departments. In the absence of such a university wide approach, however, procedures can be introduced at the faculty or department level, in which case the approach to be used is a proper subject for deliberation by the Advisory Board. Essential to whatever approach is to be adopted is the involvement of the student body. If this is to be effective students must be confident both that their responses to any questionnaire are assured to be anonymous and that anything they might say in a student-staff committee meetings, or in general conversation, will not lead to discrimination or victimisation. This confidence is most likely to be engendered if the system for teaching quality assurance is a university system applied to all faculties, and so has the backing of the most senior staff.
