Previous Page Table of Contents Next Page


Section III: Assessing Programme Design


Assessing ......

  • The macro-environment

  • The micro-environment

· Programme design

  • Sustainability

By now you should have accumulated sufficient information to assess the environment within which your programme functions. Some of the actions we have proposed are outside the remit of the programme, but nonetheless essential to ensure a positive impact of the programme on the nutritional status of the population it covers. We come now to an examination of the programme itself. There is a considerable body of literature on the methodology of programme design, which this section will not attempt to replicate. Rather the intention here is to highlight those important aspects of programme design that have been found to have a significant impact on programme performance. In this section you are asked to examine the programme design to assess the following:

1) Assessing programme relevance

Assessing Programme Design

· Programme relevance

  • Programme interventions

  • Community activities

  • Community mobilizers

  • Programme management

  • Monitoring and evaluation

  • Programme linkages

To assess programme relevance, we need to discover whether the causes of malnutrition were investigated and whether the programme’s objectives address the identified causes. Ideally you need to talk to the persons who formulated the programme, or if this is not possible, to the senior programme staff, and examine the programme document and relevant files to seek answers to the following questions:

· Was any form of problem analysis undertaken before the programme was designed? If so, is there evidence that a participatory approach was used for the problem analysis? How specific are the causes of malnutrition identified by the problem analysis?

Broad causes of malnutrition like ‘poor diet’ are not especially helpful: you need to know in what way the diet is poor. You also need to know why the diet is poor. Answers to these questions should have guided the nature of the interventions undertaken by the programme.

Planning the programme (problem analysis, selection and design of interventions) should have taken place in a participatory fashion, working closely with the programme’s target communities. If it did not, not only are errors in analysis and design more probable, but also the target communities are unlikely to have any sense of ownership or involvement.

· Does the programme have clear, realistic, relevant, measurable and time-bound objectives? Are there objectives relating to:

- Nutrition? If so, are they in line with the causes of malnutrition identified in the problem analysis?

- Community participation?

- Improving the macro and micro-environment?

- Capacity building?

Ultimately all nutrition programmes must aim to improve nutritional status (reduce wasting, stunting, obesity or micronutrient deficiencies, as appropriate), and this improvement must be measurable using accepted indicators, such as anthropometric status and/or biochemical indicators of micronutrient status. Improving nutritional status must be the primary goal or objective of any comprehensive, national nutrition programme.

Many programmes set objectives that relate only to one or a few of the impediments to improving nutritional status. Thus, a programme may aim to improve nutritional knowledge. This must be recognized as an intermediate objective only; it addresses only one impediment to improving nutritional status. Improving nutrition knowledge will only lead to better nutritional status[23] if it is accompanied by, for example, improved child feeding practices, better access to food, and activities to reduce morbidity, as appropriate. The primary objective of improving nutritional status must therefore be accompanied by a set of intermediate objectives that address the specific causes of malnutrition in the programme’s catchment area.

A nutrition programme will benefit by setting also objectives relating to community participation, improving the macro and micro-environment, and building capacity. Setting such objectives will ensure that these important issues are not forgotten during the implementation of the programme.

All objectives need to be relevant, specific, measurable, realistic and time-bound[24]. It is against its objectives that a programme’s success or failure is evaluated. Examine the programme document to determine whether it has such objectives, bearing in mind your findings of the assessments of the macro and micro-environment.

· Are programme activities targeted to specific households, communities or areas? If so, is there any undercoverage or leakage related to targeting?

Targeting can take the form of geographical targeting (to depressed areas, or areas with particular agricultural or climatic problems), of socio-economic targeting (to low income areas of cities, or to households that fall below a specified poverty line, poor elderly people or landless households), or of vulnerable group targeting (to weaning aged children, single mothers or elderly people for example). If the programme is targeted, find out if the system is working, or whether the procedures for the selection of participating communities or households are too elaborate or open to manipulation.

On targeting

“Targeting can take various forms: geographic targeting, such as found in the Kenya project (arid and semi arid lands) or the Honduras project (an area with a specific environmental problem); vulnerable area targeting [also a form of geographical targeting] to areas where the density of poor communities is high; or socio-economic targeting (to households below a poverty line, such as in Sri Lanka’s programme). In the Mexico programme a more elaborate dual form of targeting was employed: vulnerable area targeting to select programme localities, then socio-economic targeting to select participating households.

Good targeting can save resources, but elaborate screening can entail high administrative costs (for example, the case of Mexico). They are also open to political manipulation and corruption.”

2) Assessing programme interventions

Assessing Programme Design

  • Programme relevance

· Programme interventions

  • Community activities

  • Community mobilizers

  • Programme management

  • Monitoring and evaluation

  • Programme linkages

Programme interventions must address the causes of malnutrition identified by the problem analysis. They need to meet the objectives of the programme, within the specified time-frame, and be as cost-effective as possible, bearing in mind issues of equity. Costs borne by the communities, in terms of both material resources and time, must be considered. You should consider too that the most cost-effective programme is not necessarily the most sustainable, nor is a programme that is cost-effective in the short-term, necessarily cost-effective in the longer term. Consider a clinic-based vitamin A supplementation programme, for example, in comparison with a food-based approach that attempts to improve consumption of vitamin A rich foods. The supplementation programme will be more cost-effective in the short term, but the food-based approach will not only be more sustainable, but in the long run may prove less costly[25]. To assess the appropriateness of programme interventions, obtain answers to the following questions:

· Were appropriate interventions selected and implemented?

There is now a considerable body of literature on experiences of nutrition interventions worldwide. This presents an excellent resource, and should be consulted at the programme formulation stage, to help select the most appropriate and cost-effective programme design. Interventions must not only address the causes of malnutrition in the programme area, they must also be based on the most up-to-date scientific knowledge (see footnote 19), they must engage all appropriate sectors, and they must take into account local resources, conditions, food availability and cultural practices.

· Are the interventions in line with the basic nutrition services offered in the programme area?

You should check that the interventions undertaken by the programme do not contradict the services that are offered to the communities. An example of this would be a programme that distributed infant formula, while the local health center attempts to promote breast-feeding. The programme should reinforce rather than undermine the activities of the local health and agricultural authorities, provided these are indeed appropriate.

3) Assessing community activities

Assessing Programme Design

  • Programme relevance

  • Programme interventions

· Community activities

  • Community mobilizers

  • Programme management

  • Monitoring and evaluation

  • Programme linkages

This sub-component asks you to assess the extent of community participation, whether staff have been adequately trained in the participatory approach, and how it has been used and supported in the implementation of community activities.

Community participation in nutrition programmes is now accepted as a key prerequisite for success and sustainability. The aim of the participatory approach is to assist communities to become more self-reliant, with the capacity to analyse their own food and nutrition situation, identify their needs, plan activities to address these needs, secure funding and technical expertise, and implement and manage the activities. Achieving a fully participatory approach, whereby communities have a true sense of ownership of the programme, demands considerable investment of time and resources. However, once achieved, it can be maintained at little cost to the programme, provided that communities continue to have access to adequate basic services (health, nutrition, agricultural extension services, for example), technical expertise to help in their selection and design of activities, and funding support for their activities.

Participation ranges from passive participation to full community mobilization. To assess the degree to which the programme has achieved a participatory approach, you are asked firstly to assess where in this range the target communities of your programme fall. Secondly, you are asked to assess whether programme staff are adequately trained in the participatory approach.

· To assess the level of participation achieved by the programme, optimally you should visit a representative range of communities, observe community meetings in progress and have discussion with community leaders. Then situate the communities within the following range:

Levels of Community Participation



Participation


Level

Characteristics



1. Passive

People are told what is going to happen, or participate by answering questions only.



2. Consultative

People express their views, which may be taken into account, but have no share in decision making.



3. For material

People participate in activities in order to receive food, incentives cash or other incentive. Still no decision making, and participation often ends when incentives end.



4. Functional

People form groups and carry out activities to meet objectives of project, but no involvement in choosing objectives, and minimal involvement in choosing activities. Some groups may in time become stronger and more self-reliant.



5. Interactive

People participate in joint analysis and planning, joint decision-making, with project staff.



6. Self-mobilization

People take initiatives independent of project staff. They develop contacts with external institutions to access technical expertise and funding, but retain control over decision-making.

· An additional method of assessing the extent of community participation is to measure participation in five key areas: needs assessment, leadership, organization, resource mobilization, and management. To do this we suggest you use the ‘spidergram’ approach described in S. Rifkin and P. Pridmore: “Partners in Planning” (see Annex 2 for full reference). Details of how to use this method are reproduced in Annex 4.

On Community Participation

“A community-based nutrition programme is not necessarily one that employs a participatory approach... Few people-oriented programmes today will dare not to mention community involvement in some form ... [Yet] few are truly participatory in nature, engaging communities in decision-making and the selection of activities to answer their felt needs.”

Quoted in FAO’s in-depth study of nine programmes (2002)

· To assess the adequacy of training in community participation received by programme staff, and their understanding of the approach, organize a few focus group discussions with programme staff from different levels and different regions covered by the programme, to ask the following questions:

Based on your observations and discussions, you should begin to have a good idea of the programme staff’s perceptions of community participation. This information will help you decide whether additional training is needed.

· To assess the extent to which the programme has encouraged community organization, self-reliance and empowerment, obtain information through field visits, observation and discussions. On field visits ask to see community plans and evidence of activities in progress, and participate in community meetings. It is very important to have discussions with a range of community members, and not just with the leadership. Answer the following questions:

Harnessing local cultural practices and knowledge

FAO’s in-depth study of nine programmes (2002) found that:

Communities possess a wealth of knowledge and experiences that can be harnessed to become a part of the programme. In Honduras, the project was built around a soil conservation technique developed in the local village of Quesungual. Communities that have lived with food insecurity have developed a number of coping strategies to improve their chances of survival. Many of these can be retained and supported. Local cultural practices, such as Zunde raMambo [Chief’s Granary: a plot of land is farmed by the community, with the produce stored and distributed to poor families when needed] in Zimbabwe, offer entry points and strategies that are more acceptable to communities than unfamiliar, externally-imposed strategies. In Kenya the tradition of women’s groups and regular community meetings was used by the project to encourage participation.

4) Assessing the effectiveness of community mobilizers

Assessing Programme Design

  • Programme relevance

  • Programme interventions

  • Community activities

· Community mobilizers

  • Programme management

  • Monitoring and evaluation

  • Programme linkages

Community mobilizers[26] are an essential part of any project that employs a participatory approach. Some are paid by the programme, by local authorities or by the communities themselves. Others are volunteers, generally serving their own community only. Paid mobilizers are often responsible for a number of communities. Some programmes also have supervisors who monitor the work of a number of mobilizers or volunteers. Whatever the system, community workers, paid or unpaid, play a crucial role in community development and programme delivery. The success or failure of the programme relies heavily on their performance. Unfortunately, their selection, training and supervision are often given scant attention. Broadly, mobilizers must have strong technical support and supervision, but must also be accountable to the community they serve.

A method to identify community mobilizers

“[In Thailand]...mobilizers are identified and recruited as a result of a sociogram process, where the individual members of a cluster or neighbourhood in a community are asked whom, among their neighbours, they find trustworthy, someone that they tend to consult when they need advice about a particular problem. These individuals can be recruited to act as “resource” persons for their 10 or so households*. These volunteers have a relationship of trust with the households, so that their involvement in addressing problems of nutrition is an extension of their natural disposition.”

Quote from G. Nantel and K. Tontisirin (2002)

*The optimum ratio of households to mobilizer will vary according to local conditions

Another aspect that needs attention is the career aspirations and expectations of community workers. We tend to assume that having identified and recruited the workers, they will be willing to undertake the same work for the same pay for an indefinite period of time. Inevitably, dissatisfaction sets in and work performance falls off. There are ways around this: a basic career and salary structure, training opportunities, regular feedback to highlight achievements, public recognition and awards[27], and additional responsibilities. Field visits for observation are essential to assess their role and effectiveness, as well as focus group discussions with a sample of community groups and discussions with mobilizers and with supervisors, if these exist. Here are the questions that need to be answered:

“A key constraint in the project’s performance towards attaining its objectives has been the great shortage of community support staff with appropriate training and skills and the lack of institutions able to facilitate and sustain the development processes required....This has meant that not only the community action planning process, but also project-supported activities in the agriculture and health/nutrition sectors, have been poorly conducted....

Quoted in FAO’s case study of Zambia’s project (Muehlhoff, 2001)

5) Assessing programme management

Assessing Programme Design

  • Programme relevance

  • Programme interventions

  • Community activities

  • Community mobilizers

· Programme management

  • Monitoring and evaluation

  • Programme linkages

Management problems beset a surprising number of programmes. FAO’s in-depth study of nine programmes (2002) highlights the following:

The challenge with good management is to establish a structure that promotes transparency, that defines roles and responsibilities clearly, that permits quick response and limits bureaucratic red tape but that at the same time is able to check misuse of programme resources and is also not inordinately time consuming. Perhaps one of the most important features of good management is the ability to maintain a committed and motivated staff. For this frequent feedback is needed and a recognition of achievement and good performance.

Through an examination of programme reports and discussions with programme staff at all levels, answer the following questions:

· Does the programme have an adequate management system? Is support and supervision adequate along the management line? What weaknesses can you identify?

· Is the staff committed and motivated, or is staff turnover unduly high[30]? Are there well-defined roles, job descriptions and lines of responsibility?

· Does the programme demonstrate financial transparency? Are programme resources well-utilized and monitored?

· Has management training been provided at all levels?

6) Assessing programme monitoring and evaluation

Assessing Programme Design

  • Programme relevance

  • Programme interventions

  • Community activities

  • Community mobilizers

  • Programme management

· Monitoring and evaluation

  • Programme linkages

All programme planners agree that monitoring and evaluation is an essential component of good programme design. Yet few programmes make provision for adequate monitoring and evaluation, perhaps for fear that it will become a blaming exercise for failures. This should never be the case, and to reduce the chances of this ever occurring, it is important to foresee monitoring and evaluation as an integral part of the programme design. Monitoring and evaluation should therefore be:

Monitoring and evaluation can be divided into three parts:

a) monitoring (a process management tool);

b) evaluation (to measure programme performance and impact);

c) participatory monitoring (community-based, for the community’s use).

The literature on monitoring and evaluation is extensive. Two such publications are included in the reference list in Annex 2.

a) Monitoring

Monitoring is the periodic and routine collection of information throughout the life of the programme to determine whether programme delivery is proceeding smoothly. It is first and foremost a management tool for programme staff, but also provides essential information to understand and explain the results of programme evaluation. As a management tool, it answers questions such as:

Information should also be recorded on external events that can affect programme impact. This includes events such as drought or floods, civil disorder, the state of roads and bridges. As a management tool, there are two other important aspects of programme monitoring:

Assess the adequacy of programme monitoring by obtaining answers to the following questions:

· Does the programme have a monitoring system?

· Is there a system of information flow in place? Are monitoring reports scrutinized? Is feedback provided to programme staff?

· Is there evidence of timely response to monitoring information on bottlenecks and other operational problems?

b) Evaluation

Evaluation attempts to determine and document, as systematically and as objectively as possible, the relevance, effectiveness and impact of a programme in the light of its objectives. Appropriate indicators[31] must be identified, an epidemiologist must contribute to evaluation design and a statistician to data analysis. Both qualitative and quantitative information is important components of a good evaluation system. To assess the adequacy of programme evaluation, answer the following questions:

· Does the programme have an evaluation system?

· If yes, is there a budget line for this?

· Has statistical and epidemiological advice been sought in the design of the evaluation system?

· Are the indicators in line with the programme’s objectives? Has it been implemented as scheduled?

· Does the evaluation data allow you to assess programme impact?

c) Participatory monitoring

The information provided by a programme’s monitoring and evaluation system is largely of little interest to communities. What they need is a system to monitor their own progress towards achieving their own specific developmental goals. For this, it is recommended that community groups be encouraged to establish a simple system of participatory monitoring that relates closely to their own identified priorities and activities. Examples of simple community monitoring tools include:

To assess the communities’ participatory monitoring systems, answer the following questions:

7) Assessing programme linkages

Assessing Programme Design

  • Programme relevance

  • Programme interventions

  • Community activities

  • Community mobilizers

  • Programme management

  • Monitoring and evaluation

· Programme linkages

No programme can function well in isolation. Linkages with sub-national authorities are essential to its eventual institutionalization. Linkages with other programmes and the establishment of useful partnerships can enrich a programme substantially and make it more cost-effective. Much of this information will have been gathered to address points in Section II. You are asked here to re-examine this information in order to answer the following questions:

· Has the programme established good working relations with sub-national authorities, bodies or committees?

· Does the programme have collaborative linkages with other relevant programmes?

· Has the programme established useful partnerships with active involvement of partners? Partnerships[33] with NGOs, the private sector and with research and training institutions are important to access technical expertise and supplementary funding, especially funding for community activities.

8) Summarizing the assessment of programme design and action

You now have the information you need to make an assessment of the programme design. Turn now to the Summary Report provided in Annex 1, and answer the questions in Section III. Then carry out a SWOC analysis.

Programme effectiveness or the extent to which your programme is able to improve nutritional status in the area covered by the programme will be seriously undermined if your programme design is flawed. If you judge the programme design to be poor, then you need to consider whether the programme is worth continuing, and whether you should not rather formulate a new programme. In any event, you will need to discuss your options with the programme funders, especially if the programme is externally funded.

If you have identified some weaknesses, you can take any or all of the following actions, as appropriate:

i) Re-formulate programme objectives to make them relevant, achievable and measurable;

ii) Develop the necessary conceptual frameworks to ensure that that your programme addresses the causes of malnutrition in the different areas covered by the programme;

iii) Consider targeting, or improve the system for targeting, as appropriate;

iv) Examine alternative programme designs to see if any are more cost-effective and appropriate for your situation;

v) Strengthen community participation by providing additional training to all staff and by means of effective community mobilizers;

vi) Find ways to motivate community mobilizers. Discuss with them how you can improve their job satisfaction: the problems they face in their jobs, their career aspirations, their response to the ideas of social recognition schemes and additional training opportunities;

vii) Establish clear guidelines for community-level activities, covering the following aspects: identifying suitable community action groups, building community capacity, helping communities access technical expertise and funding sources, encouraging inter-community collaboration, and the use of appropriate cultural practices;

viii) Introduce a system of monitoring and evaluation, and secure funding for it, or re-design the existing one. Establish a system of information flows, and use the information for programme management;

x) Address identified weaknesses in programme management: simplify if necessary; discuss job descriptions, responsibilities, career structures and recognition of achievements with staff and modify as needed; provide (additional) training in management to supervisors;

xi) Improve relations with local authorities; establish or strengthen links and partnerships with other programmes, NGOs, the private sector and research and training institutions, as appropriate.

FAO’s in-depth study found a number of strengths, weaknesses, opportunities and constraints in the nine programmes it examined:

Strengths:

Weaknesses:

Opportunities:

Constraints (and threats):


[23] In some unique situations, lack of knowledge is the only impediment to nutritional improvement.
[24] An acronym that may be helpful is SMART: Specific, Measurable, Achievable, Relevant, Time-bound
[25] This statement assumes that supplementation will continue to be needed ad infinitum, whereas if the food-based approach achieves a real behavioural change in dietary practices these will continue with no further inputs, or with no additional inputs that are not already provided by a good GMP programme.
[26] Various titles are used for community-level workers whose task is to work with communities, to help them organize, identify needs, plan and implement activities: popular titles are mobilizers, promoters, facilitators, community or village workers, development workers. If unpaid, they may be called community volunteers. Existing staff, such as community health workers, extension staff, can also play a mobilizing role.
[27] Thailand made good use of this approach and found it essential to the smooth running of the programme. Social recognition can take the form of badges or t-shirts, public recognition through various ceremonies, or training opportunities even outside the community. Arguably, funding spent on a social recognition programme is more effective than funding spent on salaries for the volunteers.
[28] More important than providing technical training, which of necessity will be basic, is training mobilizers to know when, where and how to seek expert assistance, and to recognize the limits of their knowledge.
[29] If paid by the programme, sustainability after the programme ends may be in doubt.
[30] High turnover does not necessarily reflect on programme management; it may be the outcome of career advancement.
[31] If child anthropometric status is selected as an indicator (and it must be if reducing malnutrition is an objective of the programme), we strongly urge you not to use clinic-based GMP data. Such data are likely to be biased because of incomplete coverage and age of attendance at GMP sessions. A final point is that GMP data provides weights only, hence levels of wasting and stunting cannot be assessed.
[32] You should ask to see evidence of the system when you visit communities.
[33] FAO’s in-depth study of nine programmes (2002) found that the partnership with the Church in Brazil’s Child Pastorate programme led to a high degree of commitment and motivation on the part of community mobilizers.

Previous Page Top of Page Next Page