Chapter Four

Evaluation of TCP Projects on Food Quality Control

I. INTRODUCTION

317. The Technical Cooperation Programme (TCP) of FAO is part of the Organization's Regular Budget, funded by assessed contributions from Member Nations. TCP projects are of short duration (not over 24 months) and are aimed at solving urgent problems faced by developing Member Nations of FAO. The projects are intended to be catalytic and to result in significant follow-up action by the recipient government. Most TCPs have budgets in the range of US$ 150 000 - US$ 350 000.

318. This is the first of the thematic evaluations of TCP projects which were first proposed to the Twenty-eighth Session of the Conference as a part of the Programme of Work and Budget 1996-97. The intention was that groups of projects would be evaluated by the Evaluation Service around particular themes, in order to draw lessons for future design of similar projects, determine achievements at project level, improve cost-effectiveness and provide a basis for accountability reporting. A mechanism was established in 1996 for funding such exercises, through a contribution of US$ 500 per approved project to a TCP evaluation account. Funds began accruing to the account in July 1996 and it was decided that projects in food control, under the technical supervision of the Food and Nutrition Division (ESN) would be an appropriate subject for study in 1997.

II. EVALUATION METHODOLOGY

319. The evaluation began with a preliminary desk study, undertaken by ESN. Subsequently, plans were developed for missions to Asia, Africa, Europe, and Latin America and the Caribbean. It was decided that 19 of the 22 projects would be visited, by missions consisting of outside experts and/or Evaluation Service officers (14 projects were visited with external consultants). Five reports (Southeast Asia; North Asia; Latin America and the Caribbean; Eastern Europe; Malta and Africa) were prepared by these missions.

320. Three projects ⁵⁷ were not visited because of distance and cost. However, desk studies were carried out for these projects by the Evaluation Service, which were reviewed and commented upon by ESN officers.

III. PROJECT PORTFOLIO AND TYPOLOGY

321. The projects chosen were those approved between 1992 and 1995. There were 22 projects in 19 countries ⁵⁸ from four regions - Europe (Estonia, Latvia, Lithuania, Malta, Poland); Latin America and the Caribbean (Brazil, Caribbean Regional, Cuba, Mexico, Uruguay); Asia (Cambodia, DPR Korea, Lao PDR, Mongolia, Vietnam); and Africa (Botswana, Burkina Faso, Tanzania, Zambia). No food quality control projects were approved during the review period in the Near East.

322. For the purposes of analysis, the projects have been divided into categories:

• General Food Quality Control Improvement (including Legal consultant);
• General Food Quality Control Improvement (without Legal consultant);
• Projects with emphasis on laboratory upgrading;
• Projects dealing with specific food control and food standards issues.

323. The projects in categories 1 and 2 are broadly similar, in that they included inputs for strengthening food control administration, inspection and laboratories. While category 2 projects may have touched on legal matters, category 1 projects included specific inputs in this area. All the category 1 projects were in Europe, in countries which had the desire to join the EU and thus had to harmonize food laws and regulations with EU requirements. Legal consultants were backstopped by the Development Law Service (LEGN). The projects in Southeast Asia also advised on legal aspects, but through the food control administration consultant. LEGN provided backstopping to one of these projects as well.

324. While most of the projects in other regions also dealt with general food quality improvement, those in Latin America and the Caribbean had a more specific focus - ranging from detection of mycotoxins in Uruguay, to cholera prevention in the Caribbean, to antibiotic and hormone residue surveillance in Cuba. The main difference between these and other projects was the more specialized nature of the international consultancies.

325. Regional differences can be noted in the project typology, with the European projects falling into category 1, Asia and Africa projects tending to fall in category 2 and Latin America projects in category 4. All the African and Asian countries assisted had low-income levels, rudimentary food control systems and needs in many areas. Accordingly, it was felt that broad assistance in a number of areas was warranted. A similar logic was applied in the category 1 projects where all countries assisted were (at the time of project approval) seeking to harmonize their food control systems with those of the EU in order to gain eventual admittance to that body. By comparison, the countries in Latin America had relatively more developed food control systems where assistance in more specific areas was desired.

IV. RELEVANCE AND DESIGN OF PROJECTS

326. Table 1 shows scores given by missions on project design and implementation, aggregated by region. The various missions found that the large majority of projects responded to priority needs to improve food control systems. FAO has considerable experience in establishing food quality control in countries over the years and this has led to creation of a pool of experienced international consultants for assignments to food control projects. FAO thus has a clear comparative advantage in carrying out this type of project.

327. TCP assistance is intended to be catalytic and at the time of project formulation there was good reason to believe that these projects would lead to further action on the part of the recipient governments. Countries which requested assistance were in the process of reviewing administrative structures and the legal basis for food control; recommendations from the projects thus had a ready audience. As part of the same effort, training in food inspection and upgrading of laboratories could have been expected to lead to improvements in national food control systems, provided the training could be applied and equipment used for laboratory investigations, the results of which could be used by the national food control system. The provision of expert advice in these areas was thus a valid programme intervention.

328. Most of the projects followed a similar design, with objectives of improving elements in the food control system (legislation, administration, inspection, laboratories) where the projects provided inputs. There was a tendency for projects to have too uniform a design in some cases; it would have been better had there been a more explicit link between the problems identified and the input/output mix chosen for particular projects. The category 3 and 4 projects tended to have better specified objectives and narrower focus.

329. Projects had a mixture of consultancy inputs (plus technical backstopping from FAO), equipment and fellowships/study tours (see Tables 2 and 3). Consultants made recommendations to recipient governments in written reports and, in many cases, organized in-country training. All projects except one had an equipment component and all had a fellowship/study tour component. Equipment components ranged from 5 percent to 54 percent of project budgets, averaging 24 percent while fellowships were from 5 percent to 40 percent, averaging 14 percent. These components were included in project budgets but not specified in great detail in the category 1 and 2 projects. Final equipment lists were generally left to be completed by the laboratory consultant. This arrangement worked well in the large majority of projects.

330. Study tours on administration and inspection were generally designed to take place in the region. This proved to be a sound strategy as the few study tours in countries outside the region of the recipient country seemed to be of limited relevance. A few projects included laboratory study tours and they are commented upon in the Implementation Section below.

V. IMPLEMENTATION

331. Most of the projects were implemented within or very close to original budgets, in many cases under the original budget. Five projects had cost overruns, ranging from US$ 1 500 to US$ 34 849, usually on the Personnel component.

332. Most projects were implemented on time or with inconsequential delays. Three projects had serious delays. In one, the delay was due to a last-minute withdrawal by a consultant for personal reasons and time needed to find his replacement. The project in another country was suspended for a year at government request because of changes in the government. In the third country, delays in receipt of equipment made a brief second phase project necessary, to complete scheduled activities beyond the statutory two-year TCP limit.

333. Tables 2 and 3 show the breakdown by project for the various budget components, by dollar value and percentage. The main expenditure item for the projects as a whole was Personnel, nearly twice that for Equipment, and nearly three times the expenditure on Fellowships. There were considerable variations within the projects, however. It is interesting to note that the most successful projects had large Personnel components, while only one project with a high Personnel component was rated unsuccessful of a total of four projects so rated. Of the five projects with international personnel components under 40 percent (the current limit for newly-approved TCP projects), one was rated good, one satisfactory and three unsatisfactory ⁵⁹.

334. Almost without exception, the quality of the consultants used in the food quality control projects was high and their output appreciated by recipient countries. Ample demonstration is given by the follow-up to consultant reports that has taken place in most countries, which would indicate agreement with the recommendations. It also indicates that most of the recommendations were sound and implementable. Due to the good familiarity of the technical division with specialized consultants, the consultancies were for the most part timely (the few exceptions are noted in the regional reports).

335. The Equipment component in projects was generally small, ranging from US$ 12 000 to US$ 128 000 in a project specifically on laboratory upgrading. For these amounts, relatively little could be bought, particularly sophisticated equipment which laboratories often wish to have. The importance of the equipment component very much depended on existing facilities. For example, in Eastern Europe there was already an abundance of food control laboratories and the equipment component was less significant, although appreciated by the recipient countries. In African countries and Lao PDR, on the other hand, important assistance was given to national laboratories to carry out basic investigations.

336. In at least two cases, equipment needs were re-evaluated during project implementation and purchases directed towards supplies and apparatus rather than new equipment. This improved the utility of the equipment component in these projects. In a few cases, equipment provided seemed to be under-utilized or duplicated existing equipment. In one case, some equipment delivered was not that specified and for one piece the instruction manual was in a foreign language not understood extensively in the country. However, this was not pointed out to FAO for corrective action. In some cases, there were considerable delays in ordering and receipt of equipment.

337. The great majority of study tours took place within the same region of the recipient country and this proved to be a sound strategy, particularly for those study tours dealing with administration and inspection. Particularly valuable study tours were implemented for Latvia, Poland, Cuba, Uruguay and Zambia. Study tours seemed to be generally well-organized, although not all were appropriate, particularly for laboratory personnel. In these, it should be ensured that what is being learned abroad can be applied at home, i.e. that necessary equipment and materials are present. Laboratory study tours to more than one country are not likely to be as effective as concentrated exposure in a single laboratory. The holding of regional training courses in one project proved not to be cost-effective. It is questionable whether study tours should "automatically" be included in all food quality control projects; in particular the costs and benefits of laboratory study tours should be carefully weighed against possibilities for in-service training.

Technical Backstopping

338. All projects included provision for technical backstopping visits by ESN officers from FAO Headquarters and these were carried out. Some technical backstopping was given by LEGN in projects where there was a specific legal component, usually in the form of commenting on reports, but a LEGN staff member served very capably as the legal consultant in Latvia.

339. The ESN technical backstopping visits took place toward the middle or end of the project and served to give the officer first-hand knowledge of the project. These visits were useful for raising the profile of the TCP project in the country and, in some cases, for resolving implementation difficulties. In a few cases (particularly in Latin America), backstopping officers were involved in training activities.

340. Consideration could be given in future projects to a more active role by ESN officers, who might act as lead consultants in projects - particularly Regional Office staff if suitably qualified. This would increase the technical contribution of ESN to projects.

VI. RESULTS

By Region

341. The main purpose in undertaking a review of TCP projects in food quality control was to determine whether the projects had a positive impact and were cost-effective. The overall picture is positive. Projects have been for the most part catalytic. With a few exceptions, funds have been well-used.

342. Table 4 includes composite scores given by missions on three categories of effects and impact: legislation, laboratories and inspection, and prospects for sustainability.

343. Overall rankings were assigned to the projects in each of the countries. Of the 20 projects ranked above, 8 were ranked as Good, 8 were ranked as Satisfactory and 4 were rated as Unsatisfactory.

344. Looking at the sample as a whole, achievements in particular subject areas can be summarized as follows:

345. Legislation: This was an important area for projects in Europe particularly, rather less so in Asia and least in Africa and Latin America. All the European countries except one were in a phase of economic transition and the projects were to provide specialized assistance in developing a new food law, which would be harmonized with requirements for joining the EU, a goal with all the countries which were assisted. In no case was there a smooth passage from the FAO assistance to the adoption of a law, but in most cases the FAO consultants provided useful inputs. In Europe, it appears that the FAO project advice was most used in Latvia, where substantial changes in the Food Law were prepared with project assistance and eventually approved by Parliament. The general thrust of advice was that laws should be simple and not contain items best covered by regulations, since the latter are easier to amend as circumstances change.

346. Assistance provided in Southeast Asia on legislation was less useful and did not follow the principle stated above, with legislation proposed by projects tending to be more detailed and spilling over into matters usually best left for regulations. Governments made little use of the advice given. However, in Mongolia the project assistance gave impetus for the eventual adoption of laws on food, quality control and standardization and protection from chemical contamination.

347. Little was done in this area in Africa and Latin America. Exceptions were Burkina Faso, where valuable assistance was given in the preparation of a food law and procedures for adoption of regulations, and Mexico, where prototype standards and guidelines for certification procedures for fruits and vegetables were developed.

348. Administrative Structures: All projects except two (Tanzania and DPR Korea second project) in Africa, Asia and Europe dealt with this issue. Except Mexico, projects in Latin America and the Caribbean did not.

349. Generally speaking, the thrust of the advice in this area was toward consolidation of functions, improving coordination and involvement of interested parties. Particularly in Africa, the advice of the TCP projects has been taken up enthusiastically in all three countries (Botswana, Burkina Faso, Zambia) that had such a component in the project and changes have been/are being implemented. The system developed in Botswana is particularly commendable and may serve as a model for countries at similar levels of development.

350. The projects had less impact (at least thus far) on administrative structures in Europe and Asia, with the exception of one country where the Food Safety Branch has recently been strengthened through re-deployment of personnel. The other countries assisted in this programme in Europe and Asia have large, overlapping bureaucracies in food control. There have been some movements toward improved efficiency in some countries but from evidence cited above, in terms of leading to improved administration, TCP projects have played a more decisive role in Africa than in other regions.

351. Strengthening Laboratories: The great majority of the projects had a component for the purchase of laboratory equipment and training. Most laboratory training was carried out in-country, but in a few cases projects included external study tours. For some projects (Cuba, DPR Korea second project, Tanzania, Uruguay), laboratory strengthening was the primary focus.

352. These latter projects had varied degrees of success, but their purpose was clear. For the more general food quality control projects, the laboratory component was not part of the main thrust of assistance.

353. Considerable differences were noted in the performance level of laboratories in various countries. In Europe, the main problem was that there were too many laboratories, most with limited capacities. Project recommendations were directed toward consolidation and strengthening and improving quality assurance programmes. In Africa and Asia, with some exceptions, laboratories were few and not very active in analyzing samples. In these countries, the training emphasis was on new techniques. In at least one case, training provided on a study tour was inappropriate because the equipment used in training was not available in the trainees' home country.

354. Only in a very few instances was equipment purchased by the TCP projects not being utilized. In most cases, the equipment purchased was appropriate. In two countries, the major purchase in the equipment budget was a vehicle, to be used for sample collection and transport.

355. The projects aimed specifically at strengthening laboratory capacity have achieved the goal of enabling the laboratories to carry out specific types of analysis, although individual reports show that food quality control was not necessarily improved in at least one case.

356. Food Inspection: This was an area of concern for nearly all the projects and again, there were considerable regional differences in the problems faced and types of assistance offered.

357. In Africa, Asia and the Caribbean Regional project, the emphasis was on training in food inspection with particular emphasis on Hazard Analysis Critical Control Points (HACCP). In some countries, particularly Botswana and Zambia, there has been follow-up through further training in subjects where programmes were initially developed by the projects. This has happened rather less in Asia and in the Caribbean project. In addition, training sessions in those regions under the projects tended to be shorter than those in countries where follow-up has taken place.

358. In Europe, inspection activities are carried out more routinely than in other areas where projects were implemented. The emphasis in training programmes was on sampling procedures and establishing priorities for collection and analysis. Except for one country, the training programmes have continued in the countries assisted.

359. There is a need in all countries for inspectors to have a standard manual, for deciding on what and how to sample. In many countries assisted by TCP projects, there was no manual when the project began. In some (particularly in Asia), the project provided guidelines from FAO on the subject (FAO Food and Nutrition Paper 14/5 - Manuals on Food Quality Control: Food Inspection) and manuals were prepared in the local language. But in other countries (especially in Africa), there were recommendations made by FAO consultants for preparation of a food inspectors' manual but this was apparently not followed up.

VII. CONCLUSIONS AND RECOMMENDATIONS

360. It is clear that FAO has a considerable technical capacity to implement technical assistance projects aimed at improving food quality control. It has developed under the Regular Programme a considerable number of useful manuals and publications on the subject and made over the years strong contacts with international experts who can be drawn upon for consultancy assignments.

361. TCP projects are by nature small and therefore cannot be expected to resolve all food control problems in a member country. However, it is reasonable to expect that a project will be catalytic and promote further action in well-defined areas. In most cases, there has been an effort to address the recommendations made by consultants and backstopping missions, which throughout this report has been referred to as "follow-up". Unfortunately, there were few examples of other donors funding projects in food quality control after the TCP projects; the only clear example was Zambia.

362. Given the strong support voiced for the Codex in various FAO fora and the lack of project follow-up, it is recommended to consider whether there should be a Trust Fund established for assisting governments in adopting Codex standards in various ways, as a means for ensuring more follow-up to TCP projects on food quality control.

363. All projects studied used international consultants to render advice to governments - in legislation, administration, inspection or laboratories. The vast majority of the consultants performed well and made valuable suggestions. For general strengthening of a food control system, it is unlikely that such a project could be implemented without appropriate consultants. Countries prefer consultants who have a good familiarity with food control systems (or laws) that the recipient country would like to emulate. This fact very much decreases the possibilities for using national consultants in such projects, or international consultants without experience in countries with well-developed programmes.

364. Looking at the sample of projects as a whole, the implementation and results can be deemed successful and it can be generally recommended that FAO should continue to implement food quality control projects with TCP funding in the future, although with more focus on projects dealing with specific food control issues and less of the general type of project. This being said, the evaluation pointed to a few areas where there could be improvements.

365. Areas for consultancies were well-defined and useful in almost every case (the few exceptions were due to the quality of consultant rather than the need for the services). However, in many of the category 1 and 2 projects, there was overlap of recommendations, particularly between consultants on food control administration and food inspection, although sometimes also with laboratory consultant's reports. It is recommended that economies be achieved by combining these consultancies in some cases, or replacing the administration consultant with a technical officer from ESN as was mentioned above.

366. Practical issues often dictate when consultancy visits can take place. This said, it is desirable in TCP food control projects that multiple visits on the same subject matter in countries should be appropriately spaced. In some countries, there was room for improvement in this area. It is recommended, when it can be arranged, that a "team" approach to country visits be undertaken, with various consultants in the country at the same time. This would permit better consultation and perhaps result in better coordinated, less repetitive advice.

367. While all participants appreciated the opportunity to study outside their home countries and some of the study tours were undoubtedly valuable, it is recommended that there should be greater discrimination in the use of study tours. While a few of the study tours on administration and inspection were clearly valuable, others were less obviously useful. The few laboratory study tours presented an even more mixed picture. While knowledge gained on the tour is being directly applied in one country, laboratory study tours for two others have had no impact, either because staff have been transferred or equipment is unavailable to use what was learned. Study tours should not automatically be part of food quality control projects; particularly for laboratory study tours it is recommended that such training needs be reviewed during project appraisal to ensure that methods learned can be applied in the beneficiary country.

368. The utility of in-service training also varied. In some cases, there was extensive in-country follow-up after the project with further courses in the same subject, while in others there was no follow-up after the training was given. In-service training appeared to be particularly valuable in Latin America and Africa. Follow-up to training is particularly important since food control staff trained by the project could be expected to extend knowledge gained to other services connected with food control (e.g. Customs, which will likely be inspecting imports and the Police, who may be called upon to spot certain types of violations). Only a very few projects had much involvement with industry, which would be another target group in developing food control systems. With the limited budget of TCP projects, it is questionable whether a regional training project, with many beneficiary countries and an absence of follow-up of initial training, should be pursued in future.

369. The equipment purchased for the projects is being used, so in that sense the funds were not wasted. However, it is less clear that equipment is always such a high priority that it should be a standard feature of TCP food control projects. It is recommended that, if equipment is provided, the rationale for doing so should emerge clearly from the initial assessment (as it did in projects where laboratory strengthening is the primary objective) with a stated purpose for the component, e.g. to enable the laboratory to perform mycotoxin testing. It is a question whether limited TCP assistance is justified in cases where the results of laboratory analysis are not used to identify problems or are used in an operational food control system. In many countries, there is more to be gained immediately by strengthening visual and physical inspection, rather than laboratories.

370. As was mentioned previously, projects in some cases addressed the absence of a food inspector's manual, but sometimes did not. It is recommended to give more consideration to developing simple food safety extension materials (e.g. posters) in local languages.

371. Finally, there is an issue about the legal component of the TCP projects, which was undertaken particularly in Europe and Asia. In Europe, advice focused on harmonizing laws with those of the European Union. The advice given was largely satisfactory, although countries have been very slow to revise their food laws. However, in Southeast Asia there were considerable problems with the laws as proposed by the FAO projects. In all three countries, there was a tendency to overspecify in the law matters that would best be left to regulations; on the other hand, the drafts did not specify any safeguards or consultative procedures for the issue of regulations.

372. The universal applicability of FAO's Model Food Law is open to question. It is recommended that ESN and the Development Law Service review the document and, if necessary, issue revised guidelines for the development of national food laws which are general enough to apply in any legal system. Before the guidelines are issued, it may be useful to hold an expert consultation on its contents.

Annex

MANAGEMENT COMMENTS ON EVALUATION OF TCP PROJECTS ON FOOD QUALITY CONTROL (BY THE NUTRITION DIVISION (ESN) AND THE TECHNICAL COOPERATION PROGRAMME UNIT (TCDT))

373. The evaluation of TCP projects on a thematic basis, beginning with this review of projects on food quality control, is an important step forward in improved management of the programme. The evaluation report is accurate and objective. The conclusions and recommendations made in the report are fair and have confronted important points for improving projects in the future. A number of key points are made which require action on the part of management.

374. It is recognized that the design of projects was rather too uniform and greater efforts need to be made in assessing which specific food control problems can be addressed to the greatest benefit, in view of the limited TCP resources available. While appreciating that the most successful projects have been those with specific objectives aimed at particular aspects of food quality control, it is also pointed out that funding for pre-project formulation missions, which could introduce improvements to less than satisfactory documents received from the field, is normally considered on an exceptional basis only. Greater flexibility could, however, be further exercised in the project formulation process as required.

375. Certain areas have been identified where it is possible to introduce greater cost efficiency in the implementation of these projects and these will be examined in future project designs. However, ESN expressed the view that the proposal to combine consultancies cannot be generalized because of the large difference of conditions and facilities existing in the different countries. Experts are used according to a work plan which is submitted for periodical review during the project execution. The approach followed in TCP projects is to strengthen the food quality control in a step-by-step manner, which basically includes a review of the food control system in the country and activities of assessment and training in specific areas, avoiding overlapping of events. In many cases, countries are not capable to receive at the same time a group of experts.

376. The recommendations regarding input structure including the use of international consultants have been noted and will be taken care of as appropriate on a case-by-case basis.

377. Finally, ESN feels that the use of the FAO/WHO Model Food Law has been of great assistance to many member countries and a revision of it is not warranted.

57 Brazil, Uruguay, Burkina Faso

58 There were two projects in Cuba and Zambia for administrative reasons (maximum duration of the TCP projects had been reached); substantively they were the same project. Two different projects were implemented in DPR Korea.

59 The project rated good had a considerable international expertise component, but it was provided free to the project by a donor.