5.1 Evaluation
5.2 Continuity
5.3 Spreading and sustainability
The consensus is that evaluation, like monitoring, is important because it allows lessons to be learned, leading to improvement in future projects. But like monitoring, it is not done with sufficient rigour. For example, "Many projects cannot precisely pinpoint who has benefitted and by how much or in what way. This does not indicate the projects are unsuccessful, but that existing methods of appraisal and evaluation are too limited. There is a need for more thinking on how to appraise projects for institutional development, sustainability, and environmental impact" (Butcher 1988).
Few agencies feel that they have the resources to evaluate every project, and so evaluation, particularly ex-post evaluation, tends to be biased towards projects with problems.
Also, small agencies particularly are reluctant to use the time of project staff on evaluation when they could be getting on with the next project. The larger agencies which have separate evaluation units face the difficulty that the independence of these units makes them less able to influence the operational departments.
There are several different approaches to evaluation reporting. To some extent the termination report by project staff can be considered as an evaluation, but it is often biased by the frustrations and difficulties, and tends to be more a catalogue of problems than a balanced account of performance. An ex-post evaluation either by an evaluation unit or by sub-contractors tends to be more open-minded, but the very fact that it is done after the event means that it is dependent upon records and reports, which are likely to be unwritten, incomplete, unavailable, or less comprehensive than the evaluation team would wish. The other source of information for ex-post evaluation teams is personal recollection which is notoriously inaccurate.
Informal ex-post evaluations are also frequently done as part of a study of a group of projects. This present study is one example and use has been made of many others. Evaluations of this type are particularly useful in identifying trends, and frequently-occurring good and bad factors, but being less detailed they are less effective at evaluating individual projects.
The last type of evaluation is that carried out some years after the completion of the project. The World Bank is the only agency identified which has made a formal study of projects ten to fifteen years after their completion. The observations from that study were found to be very instructive and perhaps this type of evaluation should be carried out more frequently. Most development workers are familiar with cases where past projects can only be located by the project buildings, now serving some other purpose, or by the piles of rusting machinery - which leads to the question of continuity.
Some programmes may be designed with a fixed time-span, because they have a specific purpose, and end when the purpose is achieved. Examples are FAO short TCP projects where a specialist is provided to analyse a problem, or a bigger project with many staff over several years, such as The Ethiopian Highlands Reclamation Study. This kind of project ends with the final report, and there is no question of continuing the project, although there will hopefully be follow-up action.
Another category is where there is a weakness or deficiency which is going to continue for a long time, but donor and recipient agree on a fixed-term project, hoping that some improvement will occur in the agreed time. Institution-building projects are often like this, and there may be a series of successive separate projects, for example the USAID technical assistance to the Ministry of Agriculture in Lesotho. In this type of project the agreed fixed end-point avoids uncertainty.
The best situation of all is where the donor undertakes to support a long-term programme for as long as necessary, and an example is the SIDA support to soil conservation in Kenya.
The kind of project which is most likely to have problems and poor performance is where the objectives are too ambitious for the project life, so there is a built-in option to extend it if targets are not achieved. Experience suggests that this is likely to result in frustration and uncertainty.
The question of what happens at the end of projects was discussed under implementation in Section 4. Projects which are integrated into permanent institutions are less likely to suffer than those with a Project Management Unit (PMU) which disappears at the project end.
The problem of agricultural development is so vast compared with the available resources of aid and technical assistance that development programmes must constantly try to be catalysts for change which becomes self-spreading and self-sustaining. The ultimate objective of all aid agencies should be to make themselves unnecessary.
Sustainability is the current buzz-word in the aid industry, with major international conferences devoted to it (PANOS 1987, Conroy and Litvinoff 1988).
Some quotations from the 1987 International Institute for Environment and Development (IIED) conference follow:
Talking about sustainable spread, Chambers refers to "institutional spread", for example where activities by an NGO are taken up and adopted by other NGOs or by incorporation in government organizations; there is also spontaneous or induced diffusion, when there is a non-institutional effect such as visits by farmers who take away the seed idea which multiplies elsewhere. Chambers suggests that few NGOs or governments pay adequate attention to this. His third spreading activity is through impact on analysis and policy, quoting examples from India where projects had a big impact on official thinking.Kramer comments, "The number of projects continued by counterparts in the absence of donor funds is abysmally small". Later he asks, "When the land management techniques are known, why does widespread adoption not seem to be making much progress?" And then offers, "The answer lies in the failure of development agencies to treat communities and farmers as fully capable of achieving their own sustainable management of resources. Policies, institutions, and infrastructure all too often work at cross-purposes to sustainable local resource management".
Another issue in self-sustainability is encouraging the development of informal organizations as well as building up government agencies. The World Bank says, "A major contribution to Sustainability came from the development of grass-roots organizations, whereby project beneficiaries gradually assumed increasing responsibilities for project activities during implementation, and particularly following completion".
The World Bank 10th Audit Report discusses failures of Sustainability in a study of 25 projects which were assessed as successful at completion. Thirteen of these (50 percent) were found to have failed between 5 and 15 years after the end of the project. Some reasons were:
- Failure to build institutional capacity to continue.- Long-term sustainability was not considered during the project.
- Grass-roots organizations were not developed.
- Inadequate plans to hand over from a PMU.
- No capacity to maintain and update the improved technology which needs a continuing research base.
- Project attempted to change existing social/cultural behaviour.
- Borrower policies may change and undermine sustainability.
- Difficulties of recurrent cost financing.
