Choice of the most suitable model
The choice of evaluation model will depend largely on the information available on beneficiaries and possible control groups. In the case of LAPs, one of the major problems concerns the high level of uncertainty, at the start of projects, about the legal status of land tenure. This situation hinders the correct identification of possible beneficiaries and therefore of control groups at the ex ante stage of LAPs.
In view of this and given the wide range of impact evaluation models, they should be divided into three groups corresponding to the specific strategy of intervention results comparison and ability to attribute causality to programme actions and results . These are as follows:
- Models with comparison groups
- Comparative models with reference controls
- Unit models for comparison
Classic evaluation
To carry out an experimental evaluation, the group of project beneficiaries must be compared with a group with the same characteristics but which did not take part in the project; an evaluation of this type requires at least four data collections: two ex ante and two ex post.
The impact would be equal to the difference between the change that has occurred in the intervention group less the change that has occurred in the non-intervention group.
See formula and considerations
Im = CGI - CGNI
Im = Impact
CGI = Change Intervention Group
CGNI = Change Non-Intervention Group
The designated baseline or initial measurement and the evaluation ex post or final measurement are required for the experimental model.
To be able to conduct a classic experimental model, the course must be planned at the start of the project or at least before direct intervention with households. For the conceptual design of the baseline and questionnaire, the majority of variables must similarly be used equally for the evaluation ex post in order to compare both situations
Models with comparison groups
The classic experimental model or impact evaluation is not always feasible as the baseline is generally determined at the start of the project and not at the ex ante stage. It is similarly difficult to carry out owing to the high costs involved in collecting information relating to the control group. In these cases, models with comparison groups can offer a good alternative for evaluating programme impacts. The models that essentially use a comparison between an intervention group and a control group are described below.
Control types
Control types
Random controls
Populations that are potential beneficiaries of tenure regularization processes can be divided at random into an experimental group which is subject to the intervention, and random controls which are not subject to the intervention or classic experimental model.
Constructed controls
This involves comparing the populations that are the subject of the intervention with an equivalent group – called the constructed controls group – which has been deliberately separated from the intervention, in this case tenure regularization or titling. These can be groups constructed non-statistically but which must have similar conditions to the intervention group, and the results must be measured at the same times for both groups.
Comparative models with reference controls
Comparative models with reference controls use statistics from secondary sources that refer to the same subject evaluated or that contain the information for constructing the indicator or indicators for impact evaluation, such as cadastral databases, census regularization process monitoring systems, administrative registers, etc. Reference data from pre-existing standards can also be used for these models.
Control types
Control types
Controles estadísticos
Las poblaciones objeto, participantes y no participantes son comparadas y se registran las diferencias estadísticas entre ambos grupos. El grupo no intervenido surge de una construcción de estadísticas existentes, por ejemplo en un PAT se puede medir el incremento del acceso a crédito en relación al periodo grupo intervenido y estos resultados se pueden comparar contra las estadísticas de solicitud u otorgamiento de crédito en la zona para los mismos periodos.
Controles genéricos
Los efectos de la intervención entre las poblaciones objeto se comparan con normas establecidas respecto a los cambios típicos que ocurren en la población objeto. La información proporcionada mediante los controles genéricos puede ser utilizada para estimar lo que habría ocurrido sin la intervención del Proyecto. En la práctica, generalmente los programas contemplan mediciones solamente para el grupo intervenido, ello dificulta conocer los factores distorsionantes en la relación causa-efecto. El modelo que se elija dependerá de los recursos y del momento en el cual se opere la evaluación. Sin embargo, para los diferentes modelos, con excepción del modelo experimental, es necesario construir claramente los supuestos que se manejan en las hipótesis para poder realizar las comparaciones y poder atribuir una relación de causalidad entre el proyecto y los resultados observados.
Evaluación con modelos unitarios de comparación
Los modelos unitarios de comparación se refieren a aquellos en los que la comparación se realiza dentro del mismo grupo intervenido, esto se puede realizar si la intervención se llevó a cabo en momentos distintos o si los grupos intervenidos presentan características diferentes. Estas condiciones deben ser conocidas para poder tener una referencia estandarizada entre los dos grupos.
Control types
Control types
Reflexive controls
The target populations which receive the intervention are compared with one another against the measurements made before the intervention. For example, control groups can be chosen that were selected to take part in the projects and benefit from them, for example by means of a title deed, but which have not yet completed the administrative regularization process. This will guarantee that any such group complies with the same characteristics as the other programme beneficiaries, and is therefore comparable, although it has not yet obtained the expected effects, for example obtaining credit. In this case, this difference can be considered a counterfactual scenario. This model should be chosen only if the central variable of change is clearly associated with the programme intervention; this is generally the case when dealing with monopolistic services such as land titling.
Reconstructed controls within the intervention group
Two subgroups are formed within the intervention group based on the level of exposure to the intervention. For example, territories with longer interventions than others can be compared and the change established from the differences between the two. As a further example, two groups can be chosen which were selected to take part in the project, in this case to receive a title deed, but which have not yet completed the administrative regularization process. This accordingly guarantees that the group has the same characteristics as all the beneficiaries of the programme, and is therefore comparable. Furthermore, if this group has not yet benefited from the expected effects associated with titling, for example obtaining credit, it can be considered a counterfactual scenario.
- Sorce: Authors
Some recommendations
It is important to note that the model to be used will depend not only on the available information but also on current conditions in terms of resources and evaluation times. It is according to these factors that the most relevant model to be used for choosing the evaluation model should be defined.
Four specific cases are presented below, and alternative models recommended for their implementation in relation to their previous conditions are suggested.
See the recommended model to be used according to the study conditions
STUDY CONDITIONS | RECOMMENDED MODELS |
|---|---|
There is a baseline and ex post evaluation for the two groups (intervention and control) |
|
There is a baseline for the intervention group, however there is no possibility of a control group |
|
There is no baseline, however there is an ex post evaluation for two groups |
|
There is no baseline or control group, only an ex post evaluation for the same group |
|
Random controls
It should also be emphasized that to adequately select control groups, it is important to consider the possibility of spillover effects of RCT processes. For example, RCT processes in informal settlements may prompt municipalities or government programmes to invest in public infrastructures in these settlements, and these infrastructures may in turn generate an increase in land value (SL economic capital) and better access to services for households living there (SL physical capital). A larger proportion of the population will thus receive benefits, although the beneficiaries represent only a part. Following this example, the course of a control group in the same settlement, which does not receive title deeds, might not show significantly different impacts between beneficiaries and non-beneficiaries of the project. In this case, it would be incorrect to conclude that the programme did not generate substantial effects when it might actually have had more effects than expected. It is therefore important to analyse the specific features of the design and the possible effects of the programme to select a control group which will not be influenced by the development of the project in any of its phases.