We begin with an examination of the programme itself. This section focuses on those important aspects of programme design that have been found to have a significant impact on programme performance. You are asked to examine the programme design to assess the following:
|Assessing Programme Design|
To assess programme relevance, you need to discover whether:
Ideally you need to talk to the persons who formulated the programme, or at least to senior programme staff, and examine the programme document and relevant files.
Broad causes of malnutrition such as ‘poor diet’ are not especially helpful: you need to know in what way the diet is poor. You also need to know why the diet is poor. Answers to these questions should have guided the nature of the interventions undertaken by the programme.
Planning the programme (problem analysis, selection and design of interventions) should have taken place in a participatory fashion, working closely with the programme's target communities. If it did not, not only are errors in analysis and design more probable, but also the target communities are unlikely to have any sense of ownership or involvement.
Ultimately all nutrition programmes must aim to improve nutritional status (reduce wasting, stunting, obesity or micronutrient deficiencies, as appropriate) and this improvement must be measurable using accepted indicators, such as anthropometric status and/or biochemical indicators of micronutrient status. Improving nutritional status must be the primary goal or objective of any comprehensive, national nutrition programme. All objectives need to be relevant, specific, measurable, realistic and time-bound4. It is against its objectives that a programme's success or failure is evaluated.
Many smaller or focused programmes set objectives that relate only to one or a few of the impediments to improving nutritional status. For example, a programme may aim to improve nutritional knowledge. This is an intermediate objective, addressing one impediment to improving nutritional status. Improving nutrition knowledge will only lead to better nutritional status5 if it is accompanied by, for example, improved child feeding practices, better access to food, and activities to reduce morbidity, as appropriate. The primary objective of improving nutritional status must therefore be accompanied by a set of intermediate objectives that address the specific causes of malnutrition in the programme's catchment area.
A nutrition programme will benefit by setting also objectives relating to community participation, improving the macro and microenvironment, and building capacity. Setting such objectives will ensure that these important issues are not forgotten during the implementation of the programme.
Here are the questions you need to answer to assess the programme's relevance:
Was any form of problem analysis undertaken before the programme was designed? If so, is there evidence that a participatory approach was used for the problem analysis? How specific are the causes of malnutrition identified by the problem analysis?
Does the programme have clear, realistic, relevant, measurable and time-bound objectives? Are there objectives relating to:
- Nutrition? If so, do they address the causes of malnutrition identified in the problem analysis?
- Community participation?
- Improving the macro and microenvironment?
- Capacity building?
|Assessing Programme Design|
Targeting is a mechanism to ensure that the programme reaches only those beneficiaries for whom it is intended. Good targeting can reduce costs because resources are not wasted on beneficiaries who do not need the programme interventions.
Targeting can take the form of geographical targeting (to depressed areas, or areas with particular agricultural or climatic problems), of socio-economic targeting (to low income areas of cities, or to households that fall below a specified poverty line, poor elderly people or landless households), or of vulnerable group targeting (to weaning-aged children, single mothers or elderly people for example).
If the programme is targeted to specific beneficiaries, you need to find out if the programme is failing to reach the intended beneficiaries (undercoverage), or if it is including beneficiaries who do not meet the selection criteria (leakage):
Are programme activities targeted to specific households, communities or areas? If so, what are the criteria for inclusion in the programme?
If the programme is targeted, is the system working? Is the programme reaching all the intended beneficiaries? Are beneficiaries included who do not meet the selection criteria?
|“Targeting can take various forms: geographic targeting, such as found in the Kenya project (arid and semi-arid lands) or the Honduras project (focus on an area with a specific environmental problem); vulnerable area targeting (targeting to areas where the density of poor communities is high); or socio-economic targeting (selection of households below a poverty line, such as found in the Samurdhi programme). In the Mexico programme a more elaborate dual form of targeting was employed: vulnerable area targeting to select programme localities, then socio-economic targeting to select participating households.|
|Good targeting can save resources, on the other hand elaborate screening procedures entail unnecessary bureaucracy and high administrative costs (for example, the case of Mexico). They are also open to political manipulation ......... and corruption.”|
|Quote from FAO's in-depth study of nine programmes (2003).|
|Assessing Programme Design|
Programme interventions must address the causes of malnutrition identified by the problem analysis. They need to meet the objectives of the programme, within the specified time-frame, and be as cost-effective as possible, bearing in mind issues of equity. Costs borne by the communities, in terms of both material resources and time, must be considered. You should consider too that the most cost-effective programme is not necessarily the most sustainable, nor is a programme that is cost-effective in the short-term, necessarily cost-effective in the longer term. Consider a clinic-based vitamin A supplementation programme, for example, in comparison with a food-based approach that attempts to improve consumption of vitamin A rich foods. The supplementation programme will be more cost-effective in the short term, but the food-based approach will not only be more sustainable, but in the long run may prove less costly6.
There is now a considerable body of literature on experiences of nutrition interventions worldwide. This presents an excellent resource and should be consulted at the programme formulation stage, to help select the most appropriate and cost-effective programme design. Interventions must not only address the causes of malnutrition in the programme area, they must also be based on the most up-to-date scientific knowledge (see footnote 30), they must engage all appropriate sectors and they must take into account local resources, conditions, food availability and cultural practices.
You should check that the interventions undertaken by the programme do not contradict the services that are offered to the communities. An example of this would be a programme that distributed infant formula, while the local health center attempts to promote breastfeeding. The programme should reinforce rather than undermine the activities of the local health and agricultural authorities, provided these are indeed appropriate.
No programme will meet its objectives if the quantity and intensity of resources allocated are inadequate. Here are some examples:
- A one-off short course to improve nutrition knowledge is unlikely to be sufficient to change behaviour: it needs re-enforcement.
- Community mobilizers or health workers with too many families to cover cannot provide the intensive support needed to achieve change.
- Micronutrient supplementation or food supplementation needs to be provided at a sufficiently high level to achieve the intended improvement in nutritional status.
- Agricultural extension workers need to visit farmers frequently to improve food production.
To assess the appropriateness and adequacy of programme interventions, obtain answers to the following questions:
Were appropriate interventions selected and implemented?
Are the interventions in line with the basic nutrition services offered in the programme area?
Are the programme's resources sufficient to achieve its objectives?
|Assessing Programme Design|
This subcomponent asks you to assess the extent of community participation, whether staff have been adequately trained in the participatory approach and how it has been used and supported in the implementation of community activities.
Community participation in nutrition programmes is now accepted as a key prerequisite for success and sustainability. The aim of the participatory approach is to assist communities to become more self-reliant, with the capacity to analyse their own food and nutrition situation, identify their needs, plan activities to address these needs, secure funding and technical expertise, and implement and manage the activities. Achieving a fully participatory approach, whereby communities have a true sense of ownership of the programme, demands considerable investment of time and resources. However, once achieved, it can be maintained at little cost to the programme, provided that communities continue to have access to adequate basic services (health, nutrition, agricultural extension services, for example), technical expertise to help in their selection and design of activities, and funding support for their activities.
|On community participation|
|“A community-based nutrition programme is not necessarily one that employs a participatory approach. Most people-oriented programmes today will naturally mention community involvement in some form ..... However, few community-based nutrition programmes are truly participatory in nature, engaging communities in decision-making and the selection of activities to answer their felt needs.”|
|Quoted in FAO's in-depth study of nine programmes (2003).|
Participation ranges from passive participation to full community mobilization. To assess the degree to which the programme has achieved a participatory approach, you are asked firstly to assess where in this range the target communities of your programme fall. Secondly, you are asked to assess whether programme staff are adequately trained in the participatory approach.
|Harnessing local cultural practices and knowledge|
|FAO's in-depth study of nine programme (2003) found that:|
|Communities possess a wealth of knowledge and experiences that can be harnessed to become a part of the programme. In Honduras, the project was in fact built around a soil conservation technique developed in the village of Quesungual. Communities that have lived with food insecurity have developed a number of coping stategies to improve their chances of survival. Many of these can be retained and supported. Local cultural practices, such as Zunde raMambo in Zimbabwe, offer entry points and strategies that are more acceptable to communities than unfamiliar, externally-imposed strategies. In Kenya the tradition of women's groups and regular community meetings was used by the project to encourage participation|
Visit a representative range of communities, observe community meetings in progress and have discussions with community leaders and community members. Make sure you meet the most vulnerable members of the community, to assess the extent of input they have had in decision-making.
Ask the following questions:
Then situate the communities within the following range:
|Levels of Community Participation|
|1.||Passive||People are told what is going to happen, or participate by answering questions only.|
|2.||Consultative||People express their views, which may be taken into account, but have no share in decision-making.|
|3.||For material incentives||People participate in activities in order to receive food, cash or other incentive. Still no decision-making, and participation often ends when incentives end.|
|4.||Functional||People form groups and carry out activities to meet objectives of project, but no involvement in choosing objectives, and minimal involvement in choosing activities. Some groups may in time become stronger and more self-reliant.|
|5.||Interactive||People participate in joint analysis and planning, joint decision-making, with project staff.|
|6.||Self-mobilization||People take initiatives independent of project staff. They develop contacts with external institutions to access technical expertise and funding, but retain control over decision-making.|
Another method of assessing the extent of community participation is to measure participation in five key areas: needs assessment, leadership, organization, resource mobilization, and management. To do this we suggest you use the ‘spidergram’ approach described in S. Rifkin and P. Pridmore: “Partners in Planning” (see Annex 2 for full reference). Details of how to use this method are reproduced in Annex 4.
Now organize a few focus group discussions with programme staff from different levels and different regions covered by the programme, to ask the following questions:
Based on your observations and discussions, you should begin to have a good idea of the programme staff's perceptions of community participation. This information will help you decide whether additional training is needed.
You should now be able to answer the following questions:
What is the level of participation achieved by the programme?
How adequate was the training in community participation received by programme staff and their understanding of the approach?
Has the programme encouraged community organization, self-reliance and empowerment?
|Assessing Programme Design|
Community mobilizers7 are an essential part of any project that employs a participatory approach. Some are paid by the programme, by local authorities or by the communities themselves. Others are volunteers, generally serving their own community only. Paid mobilizers are often responsible for a number of communities. Some programmes also have supervisors who monitor the work of a number of mobilizers or volunteers. Whatever the system, community workers, whether paid or unpaid, play a crucial role in community development and programme delivery. The success or failure of the programme relies heavily on their performance. Unfortunately, their selection, training and supervision are often given scant attention. Broadly, mobilizers must have strong technical support and supervision, but must also be accountable to the community they serve.
Another aspect that needs attention is the career aspirations and expectations of community workers. We tend to assume that having identified and recruited the workers, they will be willing to undertake the same work for the same pay for an indefinite period of time. Inevitably, dissatisfaction sets in and work performance falls off.
|A method to identify community mobilizers|
|“[In Thailand] ..... mobilizers are identified and recruited as a result of a sociogram process, where the individual members of a cluster or neighbourhood in a community are asked whom, among their neighbours, they find trustworthy, someone that they tend to consult when they need advice about a particular problem. These individuals can be recruited to act as “resource” persons for their 10 or so households*. These volunteers have a relationship of trust with the households, so that their involvement in addressing problems of nutrition is an extension of their natural disposition.|
|Quote from G. Nantel and K. Tontisirin (2002).|
|* The optimum ratio of households to mobilizer will vary according to local conditions|
There are ways around this: a basic career and salary structure, training opportunities, regular feedback to highlight achievements, public recognition and awards8, and additional responsibilities. Field visits for observation are essential to assess their role and effectiveness, as well as focus group discussions with a sample of community groups and discussions with mobilizers and with supervisors, if these exist. Here are the questions that need to be answered:
Have mobilizers received any training in community participatory methods? In leadership and group dynamics? Are they effective in the communities?
Is there a system of accountability? If so, to whom are mobilizers accountable and does the system work?
Can they help communities secure funding for their selected activities? Do they know when and where to seek technical expertise9?
Is their workload manageable? Are they well supervised? Do they receive any feedback on their performance?
How are they rewarded? Are they paid? If so, by whom10? Is there any system of social recognition or motivation?
Are there opportunities for advancement? Is there a programme of workshops or further training opportunities, when they can also interact with other mobilizers?
|“A key constraint in the project's performance towards attaining its objectives has been the great shortage of community support staff with appropriate training and skills and the lack of institutions able to facilitate and sustain the development processes required…. This has meant that not only the community action planning process, but also project-supported activities in the agriculture and health/nutrition sectors, have been poorly conducted ….”|
|Quoted in FAO's case study of Zambia's project (Muehlhoff, 2001).|
|Assessing Programme Design|
Management problems beset a surprising number of programmes. FAO's in-depth study of nine programmes (2003) highlights the following:
The challenge with good management is to establish a structure that promotes transparency, that defines roles and responsibilities clearly, that permits quick response and limits bureaucratic formalities but that at the same time is able to check misuse of programme resources. Perhaps one of the most important features of good management is the ability to maintain a committed and motivated staff. For this, frequent feedback is needed and a recognition of achievement and good performance.
Through an examination of programme reports and discussions with programme staff at all levels, answer the following questions:
Is there an adequate management system? Is support and supervision adequate?
Is the staff committed and motivated? Are there well-defined roles, job descriptions and lines of responsibility?
Does the programme demonstrate financial transparency? Are programme resources well-utilized and monitored?
Has management training been provided at all levels?
|Assessing Programme Design|
All programme planners agree that monitoring and evaluation is an essential component of good programme design. Yet few programmes make provision for adequate monitoring and evaluation. Monitoring and evaluation should be viewed as an integral part of the programme design. Monitoring and evaluation should therefore be:
Monitoring and evaluation can be divided into three parts:
The literature on monitoring and evaluation is extensive. Two such publications are included in the reference list in Annex 2 (IADB 1997, ACC/SCN 2001).
Monitoring is the periodic and routine collection of information throughout the life of the programme to determine whether programme delivery is proceeding smoothly. It is first and foremost a management tool for programme staff, but also provides essential information to understand and explain the results of programme evaluation. As a management tool, it answers questions such as:
Information should also be recorded on external events that can affect programme impact. This includes events such as drought or floods, civil disorder, the state of roads and bridges. As a management tool, there are two other important aspects of programme monitoring:
Evaluation attempts to determine and document, as systematically and as objectively as possible, the relevance, effectiveness and impact of a programme in the light of its objectives. Appropriate indicators11 must be identified, an epidemiologist must contribute to evaluation design and a statistician to data analysis. Both qualitative and quantitative information are important components of a good evaluation system.
c) Participatory monitoring
The information provided by a programme's monitoring and evaluation system is largely of little interest to communities. What they need is a system to monitor their own progress towards achieving their own specific developmental goals. For this, it is recommended that community groups be encouraged to establish a simple system of participatory monitoring that relates closely to their own identified priorities and activities. Examples of simple community monitoring tools include:
To decide if the programme has an adequate monitoring and evaluation system, answer the following questions:
Does the programme have an adequately-funded monitoring and evaluation system?
Are monitoring reports scrutinized? Is feedback provided to programme staff?
Is there evidence of timely response to monitoring information on bottlenecks and other operational problems?
Has statistical and epidemiological advice been sought in the design of the evaluation system?
Are the indicators in line with the programme's objectives? Has evaluation been implemented as scheduled?
Does the evaluation data allow assessment of programme impact?
Have the communities designed and implemented a participatory monitoring system that relates to their specific development priorities12?
Do community members understand and use the information it provides?
|Assessing Programme Design|
No programme can function well in isolation. Linkages with subnational authorities are essential to its eventual institutionalization. Linkages with other programmes and the establishment of useful partnerships can enrich a programme substantially and make it more cost-effective. You are asked here to seek information in order to answer the following questions:
Has the programme established good working relations with subnational authorities, bodies or committees?
Does the programme have collaborative linkages with other relevant programmes?
Has the programme established useful partnerships? Partnerships13 with non-governmental organizations, the private sector and with research and training institutions are important to access technical expertise and supplementary funding, especially funding for community activities.
You now have the information you need to make an assessment of the programme design. Turn now to the Summary Report provided in Annex 1, and answer the questions in Section I. Then carry out a SWOC analysis.
Programme effectiveness or the extent to which your programme is able to improve nutritional status in the area covered by the programme will be seriously undermined if your programme design is flawed. If you judge the programme design to be poor, then you need to consider whether the programme is worth continuing, and whether you should not rather formulate a new programme. In any event, you will need to discuss your options with the programme funders, especially if the programme is externally funded.
If you have identified some weaknesses, you can take any or all of the following actions, as appropriate:
Re-formulate programme objectives to make them relevant, achievable and measurable;
Develop the necessary conceptual frameworks to ensure that your programme addresses the causes of malnutrition in the different areas covered by the programme;
Consider targeting, or improve the system for targeting, as appropriate;
Examine alternative programme designs to see if any are more cost-effective and appropriate for your situation;
Strengthen community participation by providing additional training to all staff and by addressing ways and means of enhancing the effectiveness of community mobilizers;
Find ways to motivate community mobilizers. Discuss with them how you can improve their job satisfaction: the problems they face in their jobs, their career aspirations, their response to the ideas of social recognition schemes and additional training opportunities;
Establish clear guidelines for community-level activities, covering the following aspects: identifying suitable community action groups, building community capacity, helping communities access technical expertise and funding sources, encouraging inter-community collaboration and the use of appropriate cultural practices;
Introduce a system of monitoring and evaluation, and secure funding for it, or re-design the existing one. Establish a system of information flows and use the information for programme management;
Address identified weaknesses in programme management: simplify if necessary; discuss job descriptions, responsibilities, career structures and recognition of achievements with staff and modify as needed; provide (additional) training in management to supervisors;
Improve relations with local authorities; establish or strengthen links and partnerships with other programmes, non-governmental organizations, the private sector and research and training institutions, as appropriate.
FAO's in-depth study found a number of strengths, weaknesses, opportunities and constraints in the nine programmes it examined:
Constraints (and threats):
4 An acronym that may be helpful is SMART: Specific, Measurable, Achievable, Relevant, Time-bound
5 In some unique situations, lack of knowledge is the only impediment to nutritional improvement.
6 This statement assumes that supplementation will continue to be needed ad infinitum, whereas if the food-based approach achieves a real behavioural change in dietary practices these will continue with no further inputs, or with no additional inputs that are not already provided by a good GMP programme.
7 Various titles are used for community-level workers whose task is to work with communities, to help them organize, identify needs, plan and implement activities: popular titles are mobilizers, promoters, facilitators, community or village workers, development workers. If unpaid, they may be called community volunteers. Existing staff, such as community health workers, extension staff, can also play a mobilizing role.
8 Thailand made good use of this approach and found it essential to the smooth running of the programme. Social recognition can take the form of badges or t-shirts, public recognition through various ceremonies, or training opportunities even outside the community. Arguably, funding spent on a social recognition programme is more effective than funding spent on salaries for the volunteers.
9 More important than providing technical training, which of necessity will be basic, is training mobilizers to know when, where and how to seek expert assistance, and to recognize the limits of their knowledge.
10 If paid by the programme, sustainability after the programme ends may be in doubt.
11 If child anthropometric status is selected as an indicator (and it must be if reducing malnutrition is an objective of the programme), we strongly urge you not to use clinic-based GMP data. Such data are likely to be biased because of incomplete coverage and age of attendance at GMP sessions. Moreover, GMP data generally provides weights only, hence levels of wasting and stunting cannot be assessed.
12 You should ask to see evidence of the system when you visit communities.
13 FAO's in-depth study of nine programmes (2003) found that the partnership with the Catholic Church in Brazil's Child Pastorate programme led to a high degree of commitment and motivation on the part of community mobilizers.