Do subsidized child care programmes work?
An operations evaluation in Guatemala City

M.T. RUEL

Marie T. Ruel is a nutritionist currently leading a programme on "Urban Challenges to Food and Nutrition Security" at the International Food Policy Research Institute (IFPRI) in Washington, DC, United States.

The accelerated rates of urbanization currently experienced by developing countries are a key driving force that policy-makers need to address both in the short term and in the long term (Garrett and Ruel, 2000; Ruel, Haddad and Garrett, 1999). One of the many consequences of urbanization is that women are increasingly involved in income-generating activities outside the home. For these women, and especially women heads of households, the scarcity of reliable and affordable child care alternatives may represent a major constraint for achieving household food and nutrition security.

The Hogares Comunitarios Programme (HCP) was established in Guatemala City in 1991 as a direct response to this recognized and increasing need. The government-sponsored pilot programme was designed as a strategy to alleviate poverty by providing working parents with low-cost, quality child care within their community. The programme aimed at promoting child development and at filling the existing gap in pre-school education in Guatemala. The successful pilot programme rapidly expanded to both urban and rural areas of all 22 departments of the country and by 1998 it included a total of 1 200 hogares comunitarios (community day care centres) and was providing care for approximately 10 000 children aged 0-7 years nationwide.

This article describes an operations (or process) evaluation of the HCP carried out in 1998 in the urban slums of Guatemala City (Ruel, 2001). Findings from the impact evaluation are presented elsewhere (Ruel et al., forthcoming). A brief description of the HCP is then provided, followed by a summary of key concepts in "operations research" and a description of the methods used for the evaluation. The final section summarizes key findings and provides suggestions to strengthen the programme.

The Hogares Comunitarios Programme

The HCP was designed as a non-traditional alternative to ensure the care of children of working parents in communities characterized by poverty and lack of access to other child care alternatives. In these communities, a group of parents select a woman from the locality and designate her as the madre cuidadora (caretaker mother). This woman becomes responsible for receiving into her home and caring for up to ten children under seven years of age, Monday to Friday, from 6 a.m. to 6 p.m. During their stay in the hogar, children receive care and affection, security and hygiene, as well as (breakfast, lunch and two snacks. Additionally, psycho-pedagogical activities are to be organized by the caretaker to stimulate the children's development and to encourage the formation of values and personal hygiene habits. For safety reasons, the programme limits the number of children younger than one year to one per hogar.

The programme provides three types of input when a new hogar is opened:

• basic furniture, kitchen equipment and supplies for ten children;
• initial training for caretakers;
• menus to guide the preparation of meals and snacks.

Although no specific norms or regulations exist regarding parents' contribution when a new hogar is opened, they are expected to provide equipment, time and support and, if necessary, to renovate or repair the future hogar.

On a monthly basis, the programme offers the following additional inputs:

• money to purchase food, fuel and educational supplies for the children (the equivalent of US$0.61/child/day);
• food donations from the World Food Programme (usually consisting of 20 kg [44 pounds] of maize, 4.5 litres [1 gallon] of cooking oil and 5.9 kg [13 pounds] of black beans - or 6 cans of fish);
• an incentive to the caretakers of US$3.33/child/month for their work.

Parents are expected to complement the incentive provided by the project with a contribution of US$5/child/month and to provide monthly supplies of basic items such as sugar, incaparina (a fortified cereal mixture), toothpaste, toilet paper and hand soap. The various inputs provided by the programme and the beneficiary parents, and the services provided by the programme are summarized in Figure 1.

Operations evaluation

Conceptual aspects of operations research

Operations research (also referred to as "process evaluation") is concerned with studying the processes by which programmes are implemented and interventions are delivered to intended beneficiaries. The main purpose of such an evaluation is to identify, as early as possible in the life of a programme any shortcomings in the process that may affect the effective delivery of the intervention and, as a result, its potential impact on the expected outcomes (Blumenfeld, 1985). The main goal is to generate the necessary information that will allow programme planners and implementers to design and test potential solutions to improve programme delivery and that will lead to the timely implementation of corrective actions (Ruel, Arévalo and Martorell, 1996; Adato, Coady and Ruel, 2000).

During their stay in the hogar, children receive care and affection, security and hygiene, as well as breakfast, lunch and two snacks

Operations evaluation of the Hogares Comunitarios Programme

The specific objectives of the operations evaluation of the HCP were:

• to review and evaluate the operational aspects (implementation) of the programme;
• to evaluate the quality of delivery of the interventions;
• to evaluate the level of satisfaction with, and opinions about, the programme of caretakers, beneficiary parents and social workers (programme staff directly supervising the caretakers in the field).

Midday meal at one hogar comunitario in Guatemala City

The study was carried out in all hogares operating at the time of the study in the urban slums of three municipios (townships) of Guatemala City. The methods used included the following:

• Semi-structured interviews with caretakers were held to collect information on programme implementation and operations (n = 206).
• Semi-structured eight-hour observations were undertaken in hogares to gather information on the quality of care and service delivery and the time allocation of caretakers and their helpers (n = 183): some hogares had been closed by the time of the observations and only single hogares (with a maximum of ten children) as opposed to multiple hogares with 20-30 children were included).
• Focus group discussions were held to gather qualitative information on the attitudes, opinions and level of satisfaction of the beneficiary parents, the caretakers and the social workers. Two focus group sessions were organized in each municipio for beneficiary parents and caretakers, respectively, and one focus group discussion was held with all 12 social workers responsible for the hogares included in the study.

Findings and suggestions

To place the programme in context, Table 1 presents a comparison of the sociodemographic characteristics of the sample of beneficiary mothers (n = 259) with working mothers from a random sample of 1 363 households surveyed as part of the study. In this sample, 37 percent of mothers were working (n = 505). The beneficiary mothers tended to be slightly younger, less educated, have fewer assets and live in a single room as opposed to an apartment or a house, and were much more likely to be single mothers than the working women from the random sample. On the other hand, the beneficiary mothers were more likely to be employed in factories and in the formal sector and to receive work-related social and medical benefits than the working women from the random sample. The income of the beneficiary mothers in the previous month was also 30 percent higher. Thus, the programme appears to be reaching a particularly vulnerable group of women and allowing them to engage in more formal and possibly more stable employment offering a greater number of benefits.

The programme's operations

The operations evaluation assessed whether inputs from the programme and from parents were received in a timely fashion and whether all the elements of the intervention package were delivered as planned.

Programme inputs

In general, the initial inputs from the programme - material, furniture and equipment - were received in a timely fashion and in good condition. The caretakers were also trained as planned, before opening their hogares. Most of them, however, stated that they would like to receive additional training. They expressed interest in being trained in the use of the menus and in learning how to substitute foods of similar nutritional value in order to accommodate changes in prices and seasonal availability. Although training in this area is supposed to be carried out by the social workers during home visits, the caretakers reported not having received it. The initial training provided by the programme should include at least some training in the use of the menus and food substitutions so that caretakers are better equipped to provide nutritious yet affordable meals to beneficiary children.

Some delays were reported in receiving the monthly cash transfers to purchase food and materials for the children. These delays were keenly felt by the caretakers, and affected their motivation and morale. The programme should therefore make special efforts to avoid payment delays in the future.

A concern consistently raised by caretakers was that the amount of cash transfer to purchase food was insufficient to allow them to follow the menus and ensure an adequate diet for the children. It is likely that this problem was due to a combination of factors, including real increases in food prices and the fact that caretakers tended to use the cash transfer to feed their whole family in addition to the ten beneficiary children. The caretakers reported using various approaches to address this financial constraint: using some of their own money, borrowing from their husband, buying cheaper food, buying where the prices were lower and reducing the amount of food given to the children. This last approach could have negative effects on the children's diets and could significantly reduce the programme's nutritional impact and thus it should be strongly discouraged. The programme should plan to reassess the cost of the menus on a regular basis (perhaps, annually) and adjust the amount of the monthly cash transfers on the basis of changes in food prices.

Parents' inputs

The interviews with the caretakers and the focus groups (with caretakers and beneficiary parents, respectively) both revealed a minimal level of beneficiary parents' participation in all activities related to the hogares. The caretakers also reported that parents had difficulties in fulfilling the two single requirements of the programme - paying their monthly fees on time and bringing the required supplies every month. Caretakers showed an honourable level of tolerance and flexibility with regard to late payments. The programme, however, should consider introducing a mechanism to increase parents' sense of responsibility and respect towards the caretakers, who should not face hardships because they are serving a population with scarce resources.

Conclusions on programme's operations

Overall, the programme seems to be operating efficiently. Limited parental participation is a main operational constraint and the programme should design and implement concrete activities to improve parents' interest and participation. Delays in providing the cash transfers to the caretakers should be avoided because they may affect the quality of attention provided to the children. The amount of the transfer should also be re-examined periodically and adjusted to compensate for increases in the price of food, to ensure that the quality of the diet in the hogares is maintained.

Quality of attention

The eight-hour semi-structured observations were used to assess the quality of attention provided in the hogares. It should be noted that all observational methods are subject to "reactivity", i.e. the subjects observed may modify their behaviour in the presence of observers (Arimond and Ruel, in press). Although this may have occurred during our evaluation, we are confident that this was not widespread because a number of less-than-ideal situations were observed, suggesting that there was still ample room for improvement.

Hygiene and safety in the hogar

The condition of the houses where the hogares were located was generally better than the average in the areas where the study took place (Table 2). Based on findings from the recent Demographic and Health Survey (INE et al., 1999), the sample of caretakers had more formal and better-equipped houses, greater availability of sanitation and water services and a larger number of assets than the general population living in marginalized urban areas of Guatemala City. This is largely a result of the specific criteria established by the programme that houses must meet certain minimum space and safety standards. Not all houses met all criteria, however, and the programme staff indicated that a degree of flexibility was required when targeting poor areas.

Hygiene was generally good, but various problems were encountered such as garbage on the floor, dirty dishes, loose animals, uncovered drinking water and caretakers who did not appear to be clean. Safety was also a concern, with almost 40 percent of the hogares having some potentially harmful objects within children's reach such as sharp, jagged objects, dangerous staircases, and construction materials, to name a few.

It is recommended that social workers pay more attention during their weekly visits to identify hygiene and safety problems and to help caretakers find ways to reduce risks for children.

Daily activities and time allocation of caretakers.

Operations research identifies, as early as possible, shortcomings in a programme which may affect the delivery of the intervention and its potential impact

The programme's suggested schedule of activities is illustrated on the left-hand side of Figure 2. According to this schedule, caretakers should spend their time as follows: 25 percent in cooking and food preparation, 20 percent in serving and feeding children, 14 percent in cleaning and maintaining the hogar, 23 percent in psycho-pedagogical activities, 9 percent in resting and personal care and 9 percent in "other" activities. The right-hand side of the Figure shows the results of the observations of the caretakers' time allocation.

The time spent by caretakers on food-related activities, in the general maintenance of the hogar, and in resting and self-care was on average comparable to that proposed by the programme. However, caretakers spent markedly less time in psycho-pedagogical activities than scheduled by the programme (6 percent compared to 23 percent), and a large proportion of the caretakers (40 percent) were not observed carrying out any educational activities at all during the observation period. On the other hand, they dedicated up to 21 percent of their time to taking care of the children and attending to their hygienic and general needs. The time allocated to these activities had been grossly underestimated in the programme's proposed schedule, which assumed that these activities could simply be combined with all others.

Caretakers also spent time purchasing food for the children, usually during the opening hours of the hogar. This meant that they left children either unattended, or with their own children or other family members present at the time. More than half of the caretakers left the hogar during our observations, for periods ranging from 15 minutes to four hours (average: one hour). It is unclear whether the caretakers felt more comfortable leaving the hogar because our observers were there, or whether the opposite was true and those who abstained from leaving did so because our observers were present. Irrespective of the direction of this potential bias, it is disconcerting to see that this practice was so common and seemed perfectly natural. The programme should exert stricter control on this aspect, to prevent accidents.

Interaction between caretakers and beneficiary children

In general, the caretakers tended to be affectionate with the children and responsive to their needs. They were attentive to the children when they cried or needed attention and they tended to settle conflicts peacefully. There were some exceptions, however, and approximately one-third of the caretakers were observed shouting at children, and 13 percent hit children (Table 3). Considering that these unacceptable behaviours were observed in the presence of our field workers, it is possible that they occur even more frequently in the absence of visitors. This is another point that should be specifically addressed in the training, retraining and supervision of caretakers by the social workers.

The caretakers also generally did a good job of keeping the children clean, to the extent possible, often without having spare clothes to change children into when accidents happened. The teaching of hygiene practices to children was also observed in most of the hogares, hand washing before and after meals being almost the norm. Sustained efforts to encourage mothers to help children acquire good hygiene practices are important.

The caretakers were usually actively involved in feeding the children, helping the little ones to eat and encouraging children with poor appetite to finish their meal.

The lack of psycho-pedagogical activities was by far the main weakness found in the evaluation of the quality of services provided by the caretakers. As indicated previously, they dedicated little time, if any, to these activities and when they did, they often only provided materials or a few suggestions and let the children play on their own. This problem was widespread and alternative solutions should be sought. One solution would be to hire specially trained educators who would be in charge of educational activities for a small number of hogares, following a model similar to that of the social workers.

Attitudes and perceptions

The information gathered through focus groups depicts a programme that is generally appreciated, well perceived by both users and implementers and that is unanimously seen as helping families with scarce resources, especially women heads of households and their children.

The main problems identified by each group as needing further attention are summarized below.

Caretakers

The caretakers identified concerns in two main areas: the quality of services and support from parents and from the programme.

The main quality issue identified by the caretakers was the absence of health supplies and services and of links with the health system. They deplored the fact that the programme had stopped providing basic medicines, vitamins and deworming drugs for the children and recommended that this service be revived. The issue of increases in food prices and related difficulties in following the menus was also discussed by all focus groups and was considered a threat to the quality of the programme.

By allowing parents to acquire better-paid employment, child care contributes to increasing household food and nutrition security

Probably the greatest concern reported by the caretakers was the expectation from the programme that they act as teachers and educators when they felt they did not have the time, the training or financial incentives to do so. The implications of this concern for the programme were discussed above. Also reiterated in the focus group discussions were the problems of lack of support from beneficiary parents.

Overall, the caretakers expressed that they enjoyed their work and that they perceived a great benefit from the programme because it allowed them to work at home while also taking care of their own children (or grandchildren).

Beneficiary parents

Beneficiary parents were generally satisfied and positive about the programme. They were extremely appreciative of the excellent work of the caretakers and were grateful to the programme for its assistance. They also indicated that the cost of the service was low compared to other alternatives, but that they would not be able to pay more.

They made several suggestions to improve the programme:

• changes to the schedule: adding Saturday care because the majority of parents work on Saturdays;
• the provision of health services and assistance when children are sick because most parents risk losing their job if they are absent from work for more than one day a month;
• the inclusion of more than one young infant per hogar. Although the current programme does not have the capacity to include more than one young infant, it may be worth considering an additional child care approach, or a similar type of subsidized programme to assist working parents with young infants.

Parents also expressed a concern about caretakers leaving children alone with family members younger than 14 years of age. This concern is legitimate, and merits additional discussions between programme staff and caretakers.

Social workers

The social workers' focus group unveiled a surprisingly high level of dissatisfaction with the programme, which the management should review carefully. Because the social workers play such an important role at the local level, interacting with caretakers, beneficiaries and communities, the programme cannot afford to ignore their concerns. Many of the issues raised by the social workers related to problems with their direct supervisors, and the lack of support they received from them.

The social workers' relationships with caretakers, on the other hand, appeared to be generally satisfactory and they felt appreciated. In their role as supervisors, they reported experiencing some difficulties at times when caretakers did not accept suggestions or did not comply with the programme norms.

The social workers reported having little contact with beneficiary parents, which is not surprising because parents are generally absent from the community during working hours. This, unfortunately, greatly limits communication between programme staff and parents.

Final comments

In summary, our evaluation revealed that the HCP is a carefully designed and well-implemented programme that is much appreciated by its users and its implementers. The evaluation highlighted some aspects that could be strengthened, especially the inclusion of a health component in the package of interventions, the development of a mechanism to promote greater parental and community participation, the strengthening of psycho-pedagogical activities to promote early child stimulation, and the development of an approach to cater for children under the age of one year. In addition, the programme should give serious consideration to adjusting the amount of the cash transfer to ensure that the quality of the diet of beneficiary children can be maintained.

The programme is clearly providing a great benefit to families with limited resources, and is reaching particularly vulnerable groups such as single mothers with limited resources. By facilitating their involvement in the labour market and allowing them to acquire better-paid employment in the formal sector, the programme contributes to their household food and nutrition security. Preliminary findings from the impact evaluation also indicate that the programme is having a significant impact on children's nutrient intake, particularly of key micronutrients such as vitamin A, iron and zinc (Ruel et al., forthcoming). Experience gained in the design, implementation and evaluation of this successful programme should provide guidance for similar programmes, which are increasingly in demand in Latin America and in other developing regions (Deutsch, 1998; Myers, 1995). Such programmes have great potential to contribute to the alleviation of poverty and food insecurity in marginalized urban areas and this potential should continue to be explored.

Acknowledgements

The author would like to thank the following individuals for contributing to this project: Bénédicte de la Brière, then at the International Food Policy Research Insititute (IFPRI), now at the World Bank; Kelly Hallman and Agnes Quisumbing from IFPRI; the field staff: Nora Coj and the field workers; the staff from the Hogares Comunitarios Programme: Doris López Montenegro, Eva Idee Alarcón, Marta Lidia de Montenegro and Dr Isabel Vicente; Elena Hurtado, private consultant; Humberto Mendez from the Instituto de Nutrición de Centro América y Panamá (INCAP)/Organización Panamericana de la Salud (OPS); and all participants in the focus groups. Funding for this study was obtained from the United States Agency for International Development (USAID), under the project no. FAO-0100-G-00-5020-00.

references

Adato, M., Coady, D. & Ruel, M. 2000. An operations evaluation of PROGRESA from the perspective of beneficiaries, promotoras, school directors, and health staff. Final Report. Washington, DC, International Food Policy Research Institute.

Arimond, M. & Ruel, M.T. In press. Assessing care: progress towards the measurement of selected child care and feeding practices, and implications for programs. Food Consumption and Nutrition Division Discussion Paper. Washington, DC, International Food Policy Research Institute.

Blumenfeld, S.N. 1985. Operations research methods: a general approach in primary health care. PRICOR Monograph Series: Methods, Paper 1. Washington, DC, Primary Health Care Operations Research.

Deutsch, R. 1998. How early childhood interventions can reduce inequality: an overview of recent findings. Working Paper No. 384. Washington, DC, Inter-American Development Bank.

Garrett, J.L. & Ruel, M.T. eds. 2000. Achieving urban food and nutrition security in the developing world. 2020 Vision Focus No. 3. Washington, DC, International Food Policy Research Institute.

Instituto Nacional de Estadística (INE [National Bureau of Statistics]), Ministry of Public Health and Social Welfare, United States Agency for International Development, United Nations Children's Fund, United Nations Population Fund, Demographic and Health Surveys Programme (DHS+)/Macro International Inc. 1999. Guatemala - Encuesta Nacional de Salud Materno Infantil 1998-1999. Washington, DC, Measure DHS+/Macro International Inc.

Myers, R. 1995. The twelve who survive: strengthening programs of early childhood development in the Third World. Second edition. Michigan, High/Scope Press.

Ruel, M., Arévalo, F. & Martorell, R. 1996. El Uso de la Investigación Operacional para la Evaluación de Aspectos Funcionales de Programas de Alimentación Complementaria. Estudio de caso: CARE Guatemala. Guatemala City, Institute of Nutrition of Central America and Panama (INCAP)/Organización Panamericana de la Salud (OPS).

Ruel, M.T., Haddad, L.H. & Garrett, J.L. 1999. Some urban facts of life: implications for research and policy. World Development, 27(11): 1917-1938.

Ruel, M.T. 2001. Operational evaluation of the Hogares Comunitarios Program of Guatemala. Washington, DC, International Food Policy Research Institute.

Ruel, M.T., de la Brière, B. Hallman, K. & Quisumbing, A. Forthcoming. Subsidized child care for working women in urban areas: an evaluation of the government-sponsored Community Day Care Center Program in Guatemala City. In Q. Woden, ed. Taking gender into account in social protection and poverty programs: case studies from Latin America. Washington, DC, World Bank.

summary résumé resumen