In Guatemala, as in most non-industrial countries, forests and trees provide food, shelter, employment and a wide variety of other wood and non-wood goods and services that are vital to people, especially in rural areas (Saket, Altrell, et al, 2002). One of the main objectives of the FAO-supported National Forest Inventory (NFI) in Guatemala is to assess the role of forests and trees in the people's lives by documenting the flow of goods and services from the management and use of trees and forests. To achieve this objective, consultants responsible for the collection of national inventory data complement the traditional measurements of biophysical parameters with information about human use of forest and trees gathered through interviews with local forest users in each selected site. The measurements follow the procedures specified in a field manual developed by FAO in collaboration with the partner organization in the host country.
The inclusion of interview variables in the NFI has been welcome by many actors in Guatemala's forestry sector. Another positive aspect of this initiative is that the introduction of the new FAO approach to NFIs has mobilized considerable financial and technical support to this effort from both FAO itself and from several national organizations. Moreover, the National Technical Unit is notably committed to the quality of the inventory's results, and the National Steering Committee that directs its work has been instrumental in promoting, adapting, and validating the inventory methods. As a result, the new approach to NFI in Guatemala seems to be gaining increasing acceptance in the sector. The national organizations involved have contributed with crucial infrastructure for the inventory in the form of developing a country-specific field manual, maps, imagery, and giving logistical support to facilitate the work of consultants in the field. Most of the consultants selected give a serious impression and seem quite qualified and experienced.
While there have not been any regular national forest inventories in Guatemala before, several of the consultants contracted for this effort have participated in forest inventories at the sub-national level. The new FAO-NFA approach, unlike traditional national forest inventories and assessments - which tend to focus on changing forest characteristics - requires that the consultant masters a wider variety of measurement methods, including interview methods. To carry out and interpret information from interviews with forest users in sometimes complex contexts can be a very challenging task. Fortunately, in the case of Guatemala, the highly qualified consultants and their productive relationship with their technical unit, bodes well for facing such challenges in a successful way. In this section, I make a series of observations regarding the methodological approach to interviews taken by our Guatemalan colleagues. I assess the measurement methods by discussing their validity, reliability and in the end I discuss to what extent it is possible to determine the level of uncertainty in this component of the NFA. I apply the criteria of reliability and validity to the three stages of fieldwork: (a) interview preparations; (b) carrying out interviews; and (c) interview results processing and evaluation.
Because variables estimated through interviews are necessarily filtered by the knowledge and personal interest of the interviewee as well as the circumstances under which the interview is carried out, it is important to prepare the interview in a way that minimizes the influence of such sources of random and systematic error. The selection of who to interview is a critical step for acquiring data that accurately reflects the population parameters. In rural Guatemala, this is particularly tricky task since there are a wide variety of different user groups and each group has its own interest in tree and forest resources. Given the great diversity among these groups' characteristics and interests, a representative sample of this complexity would need to include a significant number of observations in order to reflect reality.
Given this diversity, once a consultant has carried out a number of interviews with different forest users in a number of sites, the project is faced with the task of determining whether the completed interviews are significantly representative. Do we have enough information, or are additional interviews needed? Such tests may be carried out at the site level by each consultant team, but for the purposes of a national assessment, it would probably be more efficient to test for significant representativity at the country's three different strata levels.10 Section 6 below discusses the value of carrying out such tests and how they could be used to guide consultants in determining an appropriate level of interviewee sampling intensity.
The field manual that guides the fieldwork in Guatemala states that the consultants "should only carry out up to three interviews in each site" (Ramirez et al, 2002: p 34). Further, it states that "the individuals to be interviewed could be: the land owner, a government employee responsible for monitoring the resources in the area, community leader, people living or extracting resources close to the area, a local person contracted to assist the consultant in the fieldwork" (ibid: p. 32; emphasis added). The manual provides sparse advice to the consultants as to who should be interviewed in order to maximize the reliability and validity of the interviews. In that sense, the current version of the manual does not define the expectations on consultants with regards to the quality of information that they are responsible for. Section 6, provides some suggestions as to how such performance indicators could be included in the manual, or even in the field consultants' ToRs.
In spite of the manual's advice, some of the consultants in Guatemala seem to concentrate their interview efforts on land owners. One consultant team said their primary sources of information were landowners, but when they were able to identify other users they also interviewed these. They decided who to interview on a case-by-case basis, depending on the situation at hand. It is unclear how they selected the particular landowners that they did decide to interview. Another team of consultants seemed to apply a more rigorous interview routine in that they claimed to interview all landholders whose lands are located within the boundaries of the sampling plots. A member of a third consultant team explained that they tried to interview as many different local people as possible during their presence in the field, but it was unclear how the consultant determined who else to interview beside landowners.
None of the three consultant teams that were interviewed had an explicit method of how to select interviewees who were not landowners. The technical unit that supervises the consultants' work has, so far, left it largely up to the consultant to decide how to proceed with field interviews, while making it clear to them that ultimately they are accountable for delivering high quality information.
The existing interview preparation practices pose a series of potential problems regarding the reliability and validity of the data obtained. The apparently inconsistent selection of interviewees may lead to regionally clustered biases in the data, which may be hard to correct for once the data has been aggregated at the national level. Also, if the interview information on the flows of goods and services reflects mostly the land owners' perspective, excluding the users who are not landowners, this is likely to limit the validity of the information.
Two main questions determine the reliability and validity of the interviews carried out: First, how are interviews carried out (are the methods used reliable)? And secondly, do the variables included reflect the right concepts (is the instrument valid)?
The successful documentation of the goods and services stemming from trees and forests through personal interviews depends to a great extent on how interviews are conducted. The Guatemalan NFA manual provides little guidance for how interviews should be carried out in the field. One of the few guidelines states that "One should read carefully the variables to ask about for each forest product and use, and for that it is necessary to read carefully the different definitions and codes before starting each interview" (Ramirez et al, 2002: p. 34). The manual offers some sample questions the interviewer could ask but other than that contains no guidelines on how the interviews per se should be carried out.
To provide the Guatemalan colleagues with additional support in interviewing, FAO-FRA staff has translated the field manual developed by the Cameroon pilot NFA project (Branthomme, 2002). The latter has a very elaborate section that describes several excellent field methods for how consultants can go about organizing the interviews with users. Paying attention to these methods seems especially important in situations where the interviewee is illiterate, not accustomed to formal surveys or is suspicious of the motives behind the interview. Now that the Guatemalan colleagues have access to this set of methods, the greatest challenge to achieve reliable and valid results seems less related to the availability of interview methods than to the actual application of these methods in an appropriate and systematic manner. If the consultants do apply these methods and do so well, there is certainly less of concern for the reliability of the results. But even if the recommended methods are applied by most of the five consultant groups, it is hard to estimate the degree of total uncertainty unless we test the consistency and reproducibility of the results for the overall sample. A test-retest of the stability and equivalence of data could help shed some light on the reliability of the methods actually used. However, such testing will not get at the validity of the measures. For that a more thorough testing method is needed.
In the current approach in Guatemala, it is difficult, to ascertain the validity of the information obtained in each site because there is no independent validation or control measures that would indicate to what extent measures correspond to concepts we want to measure, and the degree of congruence between information obtained and the reality on the ground. While the potential problems of reliability can be addressed with a series of mechanical tests, validity is more complicated to test and would probably require a qualitative validation procedure by an independent expert team. Both these recommended tests are explained in more detail in Section 6.
One of the potential problems of validity in the Guatemalan data set has to do with the difficulty in linking interview information to the specific plot area that has been surveyed by the consultants. The difficulty consists of determining that the information captured through interviews actually reflects the use that corresponds exclusively to the plot (as opposed to the area in the general proximity) as the interviewees seem to have difficulties in distinguishing the boundaries of this artificial area referred to by the consultants. As a result, it may not be possible to link the information from interviews to the measurements of bio-physical measurements at the plot level. It is unlikely that valid user information will be acquired at the plot level unless each interview includes a transect walk with the interviewee. However, if the interviewer is able to distinguish between the usage of products in different forest types, even though these are not harvested from the particular plot in question, the information may still have a high degree of validity at the national level.
Another potential problem related to the validity of field interviews has to do with the possible underestimation of forest use that may result from limiting interviewing to only three different actors per site, and especially if most of these are landowners. To get a better sense of how the existing interview methods affect the validity of the overall results, I suggest an independent validation of interviews in selected sites throughout the country. Such an independent study may also serve to test the reliability of methods, discussed in the section above.
After the consultants have finished the interviews in each site, they should interpret the information gathered before they code the information in the corresponding NFA forms. Because each team of consultants is likely to interpret information in their own way, there is potential risk of inconsistent interpretations influencing the reliability of the results. Even if each team is consistent in their interpretations, the interpretations of the different teams may not be equivalent to one another. The geographical clustering of measurement errors that may result from this practice may create problems of validity since these errors are likely to be correlated with other variables (Lansing and Morgan, 1971).
Once the information is coded it is handed over to the technical team for processing and evaluation. This step is referred to as the national level process for quality control. With regards to the biophysical measurements, the technical unit is committed to carry out a series of control measurements so as to ensure reliability of methods, but no equivalent test is foreseen for the interviews with users. As a consequence, it is virtually impossible to appreciate the level of uncertainty of the results of the NFA.
Considering the limited time and resources available for the NFA, it is natural for consultants to invest the minimum effort necessary to produce a satisfactory result. Under such circumstances, the definition of what is a satisfactory result becomes crucial. Unless consultants and technical unit personnel share a common understanding of a quality standard when it comes to interviews; including how many interviews should be conducted, with whom, and how; the consultants will be tempted to minimize their interviewing efforts. Such behavior can have serious consequences of for the reliability and validity of the NFA results.
On the other hand, FAO now has an opportunity to modify the existing guidelines so as to strengthen the consultants' current incentives to produce high quality interview information. It is not too late for FAO and the National Technical Unit to introduce the specific measures that would stimulate consultants to deliver more reliable, and valid information. The next section discusses alternative ways of doing that.
10 The sampling grid for Guatemala was divided into three different strata, each with a unique sampling intensity. The Northern part of the country has a total of 29 sites; the Central part has 71, and the southern part has 8, making a total of 108 sites in Guatemala (Ramirez et al, 2002, p. 6).
