In this section, I suggest possible ways of modifying the current approach that would allow the FAO to capitalize on the strengths of the program while addressing some of its most serious limitations. The idea is to present a series of modifications that would not mean a greater workload for the contractors, but if done properly, could help increase the reliability and validity of the NFA results.
The consultants should know the rules of the game and what is expected from their performance, specifically when it comes to (a) With whom and how many should they talk in each site?; (b) What interviewing techniques should they use for different scenarios?, and (c) How should they select the interviewees in each site?
For this minimum standard to have the effect of motivating consultants to be more rigorous in interviewing, it is necessary to inform the consultants how and with what criteria their performance will be evaluated. The introduction of a minimum standard does not mean that fieldwork should be more complicated. On the contrary, the overall result of the new standard will be less uncertainty for the consultants regarding how they should organize their interviews when they know what is expected from them. In addition to less uncertainty, there may be some room for simplifications in the protocol. For instance, the new guidelines can simplify the interview protocol by not insisting on the separation of the sampled plot from areas of identical land use outside the boundaries of the plot (because in practice this distinction is artificial and difficult to apply for both consultants and interviewees);
Interview preparations can be improved by guiding the consultant in how to make a sound selection of interviewees in each site. The end results are likely to be more reliable and valid if the selection procedure considers the influence of the personal interests of interviewees. The following general approach is recommended as a starting point in all sites:
This triangulation method should be introduced as part of the minimum standard. Consultants should be given detailed information about how this triangulation method could be carried out in practice. Such guidelines could involve the following series of suggestions:
In sites where there is more than one forest type, consultants are asked to carry out a triangulation of responses for each forest type. That means that for a tract with five individual landowners, each one with one and the same forest type on their lands, the consultant does not need to interview all of the landowners as these all represent a similar interest with regards to their forest use (although consultants should be encouraged to interview more than one representative from each group of users). The minimum requirement in this case would be to interview three different individuals with a distinct interest in the trees and forest resources of the particular forest type in the site. These three individuals could be
If it still is not possible to determine which are the predominant forest uses and users, the consultant should be asked to approach more people with knowledge about local forest uses who could add to and complement the information already obtained from earlier interviews. Possible individuals who could be helpful for this task include;
The consultant should strive to achieve a gender-balanced sample and at least one of the interviewees in each plot needs to be a woman. This might imply that in some plots more than three interviews will be carried out.
Finally, for the reliability of the methods it is important to carefully document who the interviewees are for each site, what interest in the resource they represent, who referred to them as reliable sources of information, how the responses of different interviewees varied for each variable, and how the consultant interpreted the potentially conflicting responses. A key aspect of this triangulation approach is the way in which interviewees are selected. The more variability that is captured in terms of the interviewees' interests in trees and forests, the more reliable the data;
There are three types of quality tests that the technical unit should be carrying out: (1) Test of representativity; (2) Reliability tests, and (3) Validity tests.
The technical team should evaluate the representativity of the sample of interviewees at the strata level. Such an evaluation could include a series of simple statistical analyses. Moving averages and F-tests can be used to get a better handle on whether the complexities of the country's forest use are captured by the measurements in the sampled sites. To be meaningful, such tests would have to be carried out continuously by the technical unit. As consultants send in processed information from interviews in each site, the technical unit would test for how new data affects the aggregate variance and moving averages of key variables at the three regional strata levels. Such a processing method would enable the teams to correct for possible under-sampling or biased sampling in the earlier sites of the project by increasing the sampling intensity of interviewees in the latter sites.
To test the reliability of the data delivered by consultants, the technical team can carry out a series of test of both the stability and equivalence in the data. The former can be tested using a test-retest method while the latter can be checked using a split-half method described in section 4.1 above. Just as the test of representativity should be carried out continuously throughout the information gathering phase, so should the technical unit also run periodic reliability tests as more field data is added on by field consultants. Such a practice will work as an early warning system should there be detectable problems with the reliability of reported information. Such tests should be carried out on the individual strata level, since we can expect the characteristics of forest user data to be rather distinct from one stratum to another.
The overall quality of NFA hinges to a great extent on the validity of the methods used, yet the validity is harder to test than both the reliability and representativity, as there are no mechanical tests available. A qualitative validation procedure seems to be the most appropriate way to gauge the validity of the NFA methods and subsequent results. An independent qualitative study would serve the purposes of (a) testing the validity of the results obtained through interviews; (b) defining the level of uncertainty; (c) complementing the existing interview effort with additional relevant information for the inventory (e.g. documenting the benefits that some of the main groups in Guatemala receive from the different forest types); and (d) strengthening the accountability mechanism between the technical unit and the consultants.
Such an independent study has already been solicited by the NFA technical unit in Guatemala as they see this as an opportunity to strengthen the internal validity of the NFI findings. The design, planning and implementation of the independent study would be a joint effort by the technical unit in Guatemala, FAO-FORM and a resource person with experience in using social science methods in documenting formal and informal uses of forest resource in Latin American countries. The total cost of the independent study should not exceed 10 % of the total cost of the national forest inventory. Minimizing the cost level would make the inclusion of independent tests of the interview component a viable option for other NFAs as well.
What would be the expected outcome from an independent validation? The primary expectation is that with the results of the study it will be possible to assess the approximate margin of error for each of the consultant's interpretations on different interview variables. With the independent study results at hand, it will be possible to discuss the magnitude of the measurement error and other possible sources of bias in the analysis of different variables. In this sense, the results of this study will not replace the data based on the interpretation of the consultant, but it will be used to assess the quality of the consultant's information.
With a carefully designed study and a well-planned selection of sites, an independent study of this sort could provide the FAO-NFA team with valuable complementary information on the changing flow of forest goods and services to different forest user groups in society. The information identified as essential for monitoring public policy (see Section 3 above) but missing in the current FAO-NFA protocol could be accounted for in such a study. When selecting the sites, the FAO-FRA team should consider the usefulness of establishing these sites as permanent monitoring sites in which the effects of changing policy and market conditions on local patterns of tree and forest use can be assessed over time and for different segments of a country's population.
These tests may very well show that the methods used in the NFA are both reliable and valid, but until such tests are actually carried out the quality of NFA information will remain unknown. The three types of tests mentioned above will make it possible to define the level of uncertainty that is associated with the NFA findings. Presenting the results of such a testing procedure is likely to improve the NFA information users' perception of the quality of the NFA products. The long term benefit of deciding to invest in such measures of quality control is that the policy makers with a more positive stance towards the quality and credibility of the NFA products are also more likely to take its information into account when they make future policy decisions.
