5. Monitoring and evaluation

Throughout or review of the effectiveness of different habitat rehabilitation techniques we have emphasized the need for better monitoring and evaluation. Our understanding of the effectiveness of different habitat rehabilitation is limited because monitoring has often not been adequately replicated spatially (i.e. number of sites) and temporally (i.e., too short) and often designed as an afterthought. Designing appropriate monitoring and evaluation programmes for stream rehabilitation will differ by project type as well as by region, geomorphology, scale, and a host of other factors. However, there are several basic steps that must be taken to design an effective monitoring and evaluation programme that will allow us to learn more about the rehabilitation techniques. In an effort to provide clear guidance on monitoring and evaluation, we provide a brief overview of steps and considerations for developing a rigorous monitoring and evaluation programme for single or multiple projects and at fine (habitat or reach) or coarse scales (watershed) and provide key references were more detailed information can be obtained. We draw heavily from Roni (2005) and refer the reader to this reference for an in-depth treatment of the concepts we discuss in this section.

A well-designed monitoring and evaluation programme is a critical component of any resource management, conservation, or rehabilitation activity. It can also help reduce the cost and increase the benefits of future rehabilitation in part by minimizing failures (Lewandowski et al., 2002). The development of a monitoring programme is best done as an integral part of the design phase of rehabilitation. Many previous studies have been of limited usefulness because they were not designed and implemented as part of the initial rehabilitation project. The objectives of individual rehabilitation programmes and projects vary, as do the objectives of monitoring programmes. Numerous decisions that need to be made in designing a monitoring programme are often interrelated with those that need to be made in developing a rehabilitation project. Thus the two should occur concurrently well before construction of the project occurs. That is not to say that retrospective studies of past rehabilitation activities are not without utility, but that most questions or hypotheses will require collection of data before and after rehabilitation.

TABLE 21
Definitions of monitoring types (adapted from MacDonald et al.,1991 and Roni, 2005) and examples of what might be monitored for a wood placement project targeting fish. Effectiveness and validation monitoring are typically the types used to habitat evaluate rehabilitation actions.

5.1 Definition of monitoring and evaluation

Before discussing steps for monitoring and evaluation it is important that we define what we mean by monitoring, as there are several types. As with the terminology of restoration, there is much confusion about monitoring terminology or types. Monitoring is technically defined as systematically checking or scrutinizing something for the purpose of collecting specified categories of data. In ecology it generally refers to sampling something in an effort to detect a change in a physical, chemical, or biological parameter. The common types of monitoring used to examine changes in aquatic habitat and biota include: baseline, trend, implementation, effectiveness, and validation monitoring (Table 21; MacDonald et al., 1991; Roni, 2005). Determining whether a rehabilitation project was implemented correctly (implementation or compliance monitoring) is an important part of understanding why it may or may not have achieved goals and objectives. Implementation monitoring is relatively straightforward, involves quality assurance and project construction management, and may be as simple as a yes-no checklist (Kershner, 1997). Effectiveness and validation monitoring, which typically focus on determining whether an action had the desired physical and biological effects (Table 21), are often much more complex, more difficult, and longer term than implementation monitoring. They are also the type of monitoring we use to evaluate rehabilitation actions and the focus of our discussion. Other types of monitoring (status and trend) may also help plan and inform evaluation of rehabilitation actions.

5.2 Steps for developing monitoring programmes

Regardless of the type, number, and scale of aquatic rehabilitation actions, there are several logical steps that should be taken when designing any monitoring and evaluation programme. These include establishing project goals and objectives, defining clear hypotheses, selecting the monitoring design, selecting monitoring parameters, spatial and temporal replication, selecting a sampling scheme for collecting parameters, implementing the programme, and finally, analyzing and communicating results (Figure 16). Many of these steps are interrelated and some steps could occur simultaneously or in a different order than presented here. For example, monitoring design depends on hypotheses and spatial scale, just as the number of sites or years to monitor depends in part on the parameters selected. The first steps are critical for designing an effective monitoring and evaluation programme and we focus our discussion on these.

Determining the objectives of the project and defining key questions and hypotheses are the critical first steps in developing a monitoring programme. Defining the key questions will depend on the overall project objectives. Evaluation of rehabilitation actions can be broken down into four major questions based on scale (e.g. site, reach, watershed) and desired level of inference (number of projects). These include evaluations of single or multiple reach-level projects and single watershed or multiple watershed-level projects (Table 22). For example, if one is interested in whether an individual rehabilitation action affects local conditions or abundance (reach scale), the key question would be: What is the effect of rehabilitation project x on local physical and biological conditions? In contrast, if one is interested in whether a suite of different project types has a cumulative effect at the watershed scale, then the key question would be: What is the cumulative effect of all rehabilitation actions within the watershed on physical habitat and populations of fish or other biota? While some actions such as riparian plantings or instream wood placement can cover multiple adjacent reaches or occur in patches throughout a geomorphically distinct reach, the initial question is still whether one is interested in examining local (site or reach scale) or watershed-level effects on physical habitat and biota.

Determining the scale of influence for physical habitat responses requires distinguishing between habitat unit, reach, and watershed-scale effects (Frissell and Ralph, 1998; Roni et al., 2003). However, for fishes and other mobile organisms, determining the appropriate scale requires differentiating between changes in local abundance and changes in population parameters at a watershed or larger scale. Most research on habitat and biota, both for rehabilitation and other ecological studies, has focused on reach scale or individual habitat units. This information is important, but uncertainty about movement, survival, and population dynamics of biota prevent these reach-scale studies from addressing watershed or population-level questions. Studies designed to assess watershed or population-level effects can provide valuable information but also face multiple challenges (e.g. upstream-downstream trends, sampling logistics; Conquest 2000; Downes et al., 2002).

TABLE 22
Overarching hypotheses for monitoring aquatic rehabilitation divided by scale and number of projects of interest (from Roni, 2005). Most appropriate study designs are listed in parentheses. BA = before-after study design, BACI = before-after control-impact, and EPT = extensive post-treatment design. Extensive design refers to a design that is spatially replicated (many study sites, reaches, or watersheds).

From the key questions and specific hypotheses will flow the other important decisions including appropriate monitoring design, duration and scale of monitoring, sampling protocols, etc. The most difficult part and the biggest shortcoming of many rehabilitation evaluation programmes is the study design. As noted in our review of riparian rehabilitation, lack of preproject data, adequate treatments and controls, reference sites, and various management factors have limited the ability of many studies to determine the effects of rehabilitation actions. There are many potential study designs for monitoring single or multiple rehabilitation actions. None is ideal for all situations and each has its own strengths and weaknesses. Hicks et al. (1991) distilled these possibilities down to a handful of experimental designs based on whether data are collected before and after treatment (before-after, or post-treatment designs) and whether they are spatially replicated or involved single or multiple sites (intensive or extensive). They also described the pros and cons of each approach (Table 23). Many variations of these basic study designs have been used or proposed in monitoring of land use, pollution, and habitat alterations (e.g. Johnson and Heifetz, 1985; Walters et al.,1988; Bryant, 1995) and can easily be modified for use in evaluating rehabilitation actions. However, most of these modifications can be classified as either before-after or post-treatment study designs. The first include collection of data before and after implementation of the rehabilitation project often with a control reach or watershed (before-after control-impact or BACI design) and the later are retrospective studies implemented after rehabilitation and rely on comparing treated areas to suitable control (same but no treatment) or reference (ideal or natural conditions) areas. The many strengths and weaknesses of different designs are thoroughly reviewed in Hicks et al. (1991), Downes et al. (2002), and Roni (2005) (Table 23). No one design is correct for all situations-the key questions and hypotheses will help determine the most appropriate design.

Determining which metrics and parameters to monitor and measure logically follows defining goals and objectives, key questions and hypotheses, definition of scale, and selection of study design. Selecting parameters also goes hand in hand with spatial and temporal replication and sampling schemes discussed below. Parameters and metrics should not be selected arbitrarily or simply because they were used in other studies. Monitoring parameters should be relevant to the questions asked, strongly associated with the rehabilitation action, ecologically and socially significant, and efficient to measure (Downes et al., 2002; Bauer and Ralph, 2001; Kurtz et al.,2001). For example, monitoring of riparian rehabilitation will likely be focused on indicators of plant growth and diversity as well as some channel features, while instream habitats improvement may focus on instream habitat features and changes in fish numbers or diversity. Moreover, to be useful the parameter must change in a measurable way in response to treatment, be directly related to resource of concern, and have limited variability and not likely be confounded by temporal or spatial factors (Conquest and Ralph, 1998).

TABLE 23
Summary of advantages and disadvantages of the major study designs used to evaluating stream or watershed rehabilitation or habitat alteration (modified from Roni, 2005). Intensive study design generally includes sampling at one or two study sites or streams, extensive at multiple study sites, streams, or watershed. Years of monitoring needed to detect a fish response are general estimates based on juvenile salmonid studies and extensive study designs assume more than 10 sites are sampled (space for time substitution) thus fewer years of monitoring are needed.

NA = not applicable

The appropriate parameters to monitor will differ by types of rehabilitation as well as specific hypothesis. The choice of a parameter should in part be based on the different sources of spatial and temporal variability associated with that parameter. Both observation error and natural variability of a quantity will reduce the precision with which the mean of the quantity is estimated. For example, electrofishing and snorkeling are both used to estimate juvenile fish densities in small streams. While electrofishing may have a smaller observation error, it is more time consuming and thus leads to fewer surveyed habitat units. If the variability of fish is high between units then the marginal reduction in observation error may have a relatively small effect on the precision of the mean density estimate when compared with the increase in precision from snorkeling more units. Moreover, temporal variation within sites and across sites can affect the usefulness of an indicator or parameter for detecting local and regional trends in biota or habitat (Larsen et al., 2001). It is important to consider these different types of error when selecting monitoring parameters.

Numerous publications discuss different parameters to measure, the strength and weaknesses. Many regional protocols exist for different parameters. Parameters typically address watershed processes or physical, chemical, and biological changes. We summarize common parameters in each of these categories in Table 24 and attempt to link them to basic watershed processes as outlined in Figure 1. The reader should consult regional protocols for more information on which might be most useful in their region.

Determining the spatial and temporal replication needed to detect changes following rehabilitation can and should be established prior to monitoring. This will also help determine whether the initial parameters selected will be useful in detecting change to the rehabilitation action in questions. This can be done using relatively straightforward power analysis found in statistical software packages and statistical texts. Similarly sampling schemes for collecting data within a given study area are covered in similar texts (e.g. simple random, systematic, stratified random, multistage, double sampling, Line transect).

Once the monitoring programme has been designed and implemented, the results obviously need to be written up and published. While this seems intuitive, many studies on habitat rehabilitation have only been published as grey literature. Moreover, the published literature is likely biased towards projects that showed an improvement following rehabilitation. Rehabilitation actions are experiments and reporting both positive and negative findings are critical for improving our understanding of the effectiveness of different measures, spending limited rehabilitation funds wisely, and restoring aquatic habitats and ecosystems.

5.3 Conclusions for monitoring and evaluation

Monitoring and evaluations is critical to understanding rehabilitation actions and spending future funds wisely. Key factors to consider when developing a monitoring and evaluation programme for rehabilitation include:

• Establishing hypotheses, choosing an appropriate study design, and selecting sensitive parameters linked to the hypotheses.

• The lack of published evaluations of habitat rehabilitation emphasize the need for better reporting and publishing both successful and unsuccessful projects.

• Design of monitoring is best done as part of project design, not as an afterthought.

• The failure to detect significant changes in watershed processes, physical habitat, or biota has often been because of poorly designed monitoring and not following the steps defined above.

TABLE 24
Common parameters utilized to evaluate rehabilitation projects