7.
Dealing With Uncertainty: Sensitivity Analysis

7.1 Introduction

Mention has been made that: (1) identification and valuation of costs and benefits for any project involve looking into the future; (2) estimates of present and future values are subject to uncertainty; and (3) the analyst needs to recognize and to treat explicitly the uncertainty surrounding forecasts of future events and values. This chapter considers how to treat uncertainty in a project analysis. The main technique suggested is sensitivity analysis, or the testing of the sensitivity of the chosen measure(s) of project worth to alternative assumptions about values of inputs and outputs and various technical relationships, i.e., how the value of the NPV or ERR changes if the assumed value(s) of a given parameter (group of parameters) is (are) changed.

Uncertainty refers to the fact that the analyst cannot be sure today about anything that is going to happen in the future. Or, because of inadequate information, the analyst cannot be sure about past and present events which provide a basis for forecasting future conditions. Using available information relating to past events the analyst makes estimates (or guesses) of what is likely to happen - what the future demand for pine sawnwood will be, what the cost of labor will be, how natural hazards will affect a plantation project, etc. However, the analyst is never certain how close these estimates will be to what actually will happen. There always is some uncertainty involved.

The analyst may feel more confident about some estimates than others, particularly if there is more experience (more accurate observation of past events and trends) on which to base the estimates. In some cases the analyst may even have enough quantitative information about past occurrences to be able to estimate the statistical probability of the occurrence of some future event. This is referred to as a situation of risk. In contrast, when there is little or no basis for deriving quantifiable probabilities, there is a situation of uncertainty.

While this distinction between risk and uncertainty is useful in conceptual discussions, it may merely serve to confuse the analyst dealing with a real project, since in reality the analyst is dealing with a continuum from one extreme, where probabilities of occurrence can be quantified (e.g., in cases where actuarial evidence is available), to the other extreme where no information is available on which to base probability estimates. In most cases, the forecasting problems faced in project planning fall somewhere between situations of risk and total uncertainty.

7.2 Purpose of Treating Uncertainty

In the assessment process, the analyst identifies and then values the inputs and outputs associated with the project being analyzed (chapters 4 and 5). The resulting expected values, i.e., those considered to be most likely to occur, are then used in the initial calculation of the chosen measure(s) of project worth (chapter 6). To make a complete and useful economic analysis the analyst also has to provide some idea of what would happen to the chosen measures of project worth or efficiency if the actual values of various inputs and/or outputs turn out to be different from the expected values used in the analysis. If a reasonable change in the assumption about the expected value for a given parameter (or value of a combination of parameters) is critical in terms of the expected measure of project worth or efficiency, the analyst generally will want to take some steps to reduce the uncertainty. The term reasonable in this context refers to an estimate of what the possible values are for a parameter around the expected value used in the basic analysis. The term critical generally refers to the point where the measure of project worth or efficiency moves from positive to negative (or vice versa) in terms of the relevant decision criterion.^[32]

7.3 Guidelines for Treatment of Uncertainty

The following practical systematic approach to analyzing uncertainty is recommended. It involves three steps which are explained in more detail later:

1. identify likely major sources of uncertainty for the project being analyzed, and for each source establish some estimate of a reasonable range of values for the parameters involved;

2. carry out a sensitivity analysis for the project using various combinations of different assumptions concerning the values of the parameters associated with the major sources of uncertainty and analyze in more detail the parameters for which changes in value assumptions are critical in terms of project outcome; and

3. determine appropriate ways of changing the design of the project or modify it to eliminate or reduce the major sources of uncertainty which are critical in terms of project outcome.

Sensitivity analysis provides a low cost means to identify critical project parameters in order to design a sound, workable project, and to understand and reduce the uncertainty surrounding the project outcome. The degree to which further information is generated on critical parameters to which the project outcome is sensitive depends on the funding available, the estimated impact of uncertainty on project outcome, and other factors. The three steps outlined above provide a logical framework for the process.

7.4 Identifying Likely Major Sources of Uncertainty

Uncertainty is associated with the availability and timing of most inputs and outputs, relationships between inputs and outputs (production functions), input and output prices (or values), and even the objectives of the project. It would be difficult and expensive to deal with all the uncertainties associated with every factor involved in a project. Thus, a first step is to identify systematically the major categories of uncertainty associated with a proposed project, and initially assess their potential importance to project decision making.

In forestry projects some of the main sources of uncertainty which may be important include:

1. Natural factors such as weather, incidence of fire, insects and diseases, and natural variation between and within species grown in different locations. These elements of uncertainty are often particularly important for plantation projects since the period between investment and return (harvest) can be long. (In some cases these factors can be analyzed in terms of probabilities.)

2. Technology and productivity factors related to processing different types of wood, input-output relationships in tree growing, processing yields, effects of alternative technologies (including silvicultural systems) on nonwood values derived from forests, labor productivity, transportation systems, etc.

3. Financial and economic factors related to values assumed for inputs and outputs, availability and cost of capital, etc.

4. Human factors related to labor availability and cost, the ability to forecast future events (wood volume availability, markets, etc.), and, most important, management capability.

The potential importance of any of these sources of uncertainty will depend on the circumstances surrounding the particular project being analyzed. Theoretically, the analyst could test the sensitivity of project outcome to changes in assumptions concerning any input or output parameter or combination of such. In practice, the sensitivity analysis will be limited to a few major potential sources of uncertainty for any given project. The analyst has to use good judgement in deciding which parameter values to test in the sensitivity analysis, given time and budget constraints. If future labor values are particularly uncertain for example, and labor is an important input item in the project, then a sensitivity analysis should be carried out for alternative assumptions concerning future labor value (see previous example). Generally, the analyst should analyze the impact on project worth, or other chosen measures of economic efficiency, of changes in estimated output values, since these often have the greatest impact on project outcome. There are no rules for choosing the parameters or combinations to be tested. The FAO case studies provide examples of which items were chosen to be tested in a variety of actual forestry project situations.

In general, if an acceptable NPV and/or ERR is obtained for a project, using the initial estimates of parameter values (the expected values), then the analyst will be interested in testing alternative value assumptions that are less favorable in terms of project outcome, i.e., higher cost assumptions and/or lower benefit assumptions. The results provide some indication of how large unexpected cost increases or benefit value reductions would have to be to have a critical effect on the chosen measure(s) of project worth (see previous definition of critical).

To summarize, the analyst first assesses what the main elements of uncertainty and risk are likely to be for the proposed project. This type of assessment may identify some potential problems, i.e., delay in start-up, potential factor cost increases, wood supply bottlenecks, market uncertainties, etc. Such information provides the analyst with a first approximation of the factors which should be tested in the sensitivity analysis. The analyst then looks at the relative magnitude and timing of various input and output items (which can be identified from the value flow tables for the project being analyzed) and lists all those which represent a significant part of project benefits or costs. The analyst then makes an initial estimate of a range of values which could reasonably be expected for each, relying on past experience and projected trends. At this stage, the analyst should err on the side of making the range too broad, rather than too narrow - narrowing can occur in later stages of the analysis. The analyst also makes some estimate of the interdependence of the values of the input and output factors, e.g., the extent to which lower or higher prices for some inputs and outputs are associated with lower or higher prices for other inputs and outputs.

In practice, the analyst generally ends up with a limited number of major parameters which will be tested in the sensitivity analysis. As mentioned, the case studies cited in Appendix A provide some examples of practical sensitivity analyses for some actual forestry projects.

7.5 The Sensitivity Analysis

Using the list of parameters and estimates of the reasonable range of values for them (as developed in the previous step), the analyst then carries out the sensitivity analysis. However, if systematically organized, it is comparatively simple to carry out the analysis using a hand calculator. There are programmable hand calculators which can easily handle the complex calculations involved in a sensitivity analysis. If time permits, it is better to include a number of sensitivity analyses rather than a few, since sometimes it is not easy to anticipate the factors to which the project outcome is sensitive.

In addition to an analysis of alternative parameter values, the analyst may also want to test the sensitivity of results to delays in implementation, and to changes in assumptions which reflect different objectives. This latter type of sensitivity analysis is relevant in cases where objectives include redistribution of income, environmental quality, increased employment, etc., in addition to the economic efficiency objective.

7.5.1 Using net present value measures for sensitivity analysis

It is usually desirable to test the sensitivity of project outcome to a combination of changes in input and/or output value assumptions and different levels of changes in the values for given inputs or outputs. The sensitivity analysis can be carried out using either NPV or ERR, though NPV is most commonly used.

Table 7.1 shows the sensitivity analysis results for a fuelwood project in the Republic of Korea.^[33] Using a 12 percent discount rate, the project had a NPV of 102,500 Won/ha. The table shows the sensitivity of NPV to a 20 percent change in any of the major cost and benefit elements shown in column 1.

Table 7.1. Korea fuelwood case study - sensitivity analysis ('000 Won/ha).

Source: See Case Study no. 2. FAO (1979).
* Net present worth (NPV at 12 percent *= 102.55).

The entries in the body of the table are interpreted as follows (using planting cost as an example):

• if planting cost were 20 percent higher than expected, then the NPV (column 2) would be Won 8,400/ha lower, other assumptions remaining as before;

• if planting cost were 20 percent lower than expected, then NPV would be Won 8,400/ha higher.

In other words, the table can be used to estimate changes in NPV due to increases or decreases in the value of any given item.

In addition to these basic interpretations, estimates of sensitivity of measures of project worth can also be derived from

• Different magnitudes of changes for a given parameter value. For example, a 40 percent increase in planting cost would result in a W 16,800 (W 8,400 x 2) decrease in NPV. Similarly, a 30 percent increase would result in a W 12,600 (8,400 x 1.5) decrease in NPV.

• Combinations of changes in input/output values. For example, suppose all costs shown in the table, except harvesting cost, were assumed to be 20 percent higher. The cumulative effect on NPV would be to reduce it by W 30,270/ha, or ([14.20 + 8.40 + 2.10 + 4.07 + 1.50] x 1,000). Since the expected value of the NPV was Won 102,500/ha, the project would still be considered economically efficient since the NPV would still be positive (Won 102,500 - Won 30,270). Any other combination of changes and magnitudes of changes could be tested in the same way. Thus, it is an extremely flexible and inexpensive approach to testing the outcome of the project (NPV) to a great variety of value assumptions.

It should be noted that the table does not tell the analyst anything about the interaction between factors, i.e., which combinations and magnitudes are most likely. That still remains as a judgmental task of the analyst. But once the analyst has settled on likely combinations, s/he can assess their impacts by using the sensitivity table. Further, the effects of changes in a parameter value are assumed to be linearly related to the measure(s) of project worth (i.e., the NPV in this case).

It is recognized that in some cases an ERR is used instead of NPV as the measure of economic efficiency. A sensitivity analysis using the ERR measure involves recalculation of the ERR for each change in assumption or combination of assumptions.

The sensitivity analysis using NPV as a basis can also provide some critical information concerning sensitivity of ERR to changes in input or output parameter values. This follows from the definitions discussed in chapter 6, where it was pointed out that when NPV is zero the ERR is equal to the discount rate used in calculating the NPV. Thus, in the NPV sensitivity analysis, when costs are increased (benefits decreased) to the point where NPV is zero, then the ERR is equal to the discount rate used. This breakeven point is of interest to decisionmakers (see section 7.5.2).

If the analyst wants to test the sensitivity of ERR to specified changes in parameter values (other than those which result in a NPV of zero), the ERR must be recalculated for each change in value. If a computer is available, it is a simple matter to run through a great number of different combinations in a short time. If a desk calculator is used, it is equally simple in terms of process, but more cumbersome in terms of the time and steps involved. (It should be pointed out that there are some relatively inexpensive calculators available which can handle this type of sensitivity analysis in a short time and in a relatively simple manner.)

7.5.2 Breakeven analysis

One common type of sensitivity analysis is the breakeven (BE) analysis. Decisionmakers are interested in how much parameter values can change before an acceptable measure of project worth becomes unacceptable, whatever the criterion (or criteria) for acceptability. For example, they may want to know how much higher can costs be and/or how much lower benefits can be before the NPV drops below zero, or the ERR drops below the accepted discount rate. Similarly, for projects producing negative NPVs or ERRs below the guiding rate, decisionmakers may want to know how much costs would have to decrease, or benefits increase, in order to make the project acceptable in terms of the chosen criteria. This type of BE analysis provides useful information particularly in cases where the decision on a project will be based on a number of considerations in addition to economic efficiency.

Strictly speaking, BE analysis is usually carried out by varying the value of only one parameter, with all others held constant at their expected values. However, it also can be carried out for a general change in costs or benefits, e.g., by determining what percentage change in all costs is needed to reach the breakeven point, where NPV = zero, or ERR = the accepted discount rate.

The values of parameters being tested which make the NPV = 0 or the ERR = accepted discount rate are called switching values, i.e, the values which switch the decision on a project (based on these criteria) from a “yes” to a “no,” or vice versa.

In cases where uncertainty about future values or benefits is particularly high, the analyst can use a cost-price approach. In this case, the analyst calculates the price or value of the output which would make benefits equal to costs when both are discounted at the accepted discount rate. Thus, this is merely a variation on the basic BE analysis. The following example of calculation of cost price illustrates the approach.

The cost-price approach has further application in cases where a project involves nonmarket priced goods and services, e.g., environmental effects. It provides the decisionmaker with information on what such goods or services have to be worth if the project is going to breakeven in terms of the relevant social rate of discount. While the decisionmaker may not be able to decide on a specific value for some nonmarket priced output, it may be possible to say: “It is at least worth that much, therefore, the project is acceptable from an economic point of view.” Alternatively, if the cost-price is very high, the decisionmaker may say: “I cannot justify the value implied by the cost-price calculation. Therefore, I will not accept the project as being acceptable in economic terms and will reject it, or attempt to redesign it to reduce costs.”

7.6 Dealing with Critical Factors Identified in the Sensitivity Analysis

Where a reasonable change in the assumption about the expected value for a given parameter (or values of a combination of parameters) is critical in terms of the expected outcome of a project, it is desirable to generate additional information about the parameter(s), if this is possible.^[34]

This may involve estimating the probabilities of different values occurring using statistical sampling techniques and available data. Or, it may merely involve developing subjective probabilities, or a number of other less formal approaches to getting more information about the likelihood or occurrence of the values that are critical to the project outcome.

Technical personnel and available literature can be consulted to obtain estimates of parameter values and their ranges under varying conditions. More detailed effort can be spent on market surveys. Further, project planners often can find that a wealth of information on species characteristics and other properties of woods is available from national or international wood testing laboratories. Such information should be used to full advantage. Similar sources can provide information about biological production functions, insect and disease problems, etc. In most cases, data on which to base an objective probability analysis are lacking and cannot be generated in a short period of time. Yet, often considerable usable information is available for use in developing subjective probabilities.

If further information on the critical parameter(s) indicates that there is a reasonable chance (perhaps 1 in 20, or whatever is chosen) that the parameter(s) could indeed take on values which would influence the decision regarding a project, then the following project planning alternatives (which are not mutually exclusive) are available:

• change project design;

• build in contingencies and safeguards; and

• adjust the decision criteria used.

The first two of these possibilities are discussed below. The third relates to the broader issues surrounding project decisionmaking and is outside the scope of EAFP.

7.6.1 Changing the project design

It may be possible to reduce uncertainty by redesigning the project, e.g., changing its scale, changing factor proportions, integrating it with further processing or with raw material production. Or, flexibility may be built into the project by staging Various project activities in a different way and with a different time schedule than initially planned, or by redesigning it to include more flexibility in terms of choice of factor inputs or outputs after implementation, etc.

Some examples will help illustrate how redesign can reduce uncertainty. If an initial project design is for project at a scale that would fully meet an estimated future market demand which is somewhat uncertain, then perhaps the project can be scaled down so that its capacity is near a lower estimate of market demand. This would reduce market uncertainty effects on the project. At the same time, if economies of scale are involved, it may increase cost. In this case the project planner has to weigh reduced uncertainty against higher costs. Or, it might be possible to redesign the project to start with a smaller capacity sawmill or plantation, and gradually build up capacity in phases as estimated future market conditions, factor availabilities, etc., become less uncertain. Other project design responses to uncertainty are possible (see box 7.3 for examples).

A few words of caution are needed concerning redesign. In most cases, if the initial project design was based on thorough analysis of alternatives, then it was likely considered to be an optimum design in terms of the criteria for judging project worth and contribution. If redesign is undertaken, it is likely that expected costs will be increased and/or expected returns reduced over the initial optimum design. There is a need to consider trade-offs between lower levels of uncertainty and lower levels of project worth (as compared with the expected return for the initial optimum design project). While the project planner can attempt to calculate and point out some of the trade-offs involved, it remains a matter of judgement as to the choice between alternatives. No general rules can be made, since it is difficult to quantify a decisionmaker's subjective weighting of uncertainty.

Redesign is not the answer to all problems of uncertainty and should be approached cautiously. In many cases, redesign may not be desirable, and it will be necessary to resort to other methods of treating uncertainty. In some cases of uncontrollable uncertainty, redesign may not be possible in the context of the project objective, and other approaches may have to be used.

7.6.2 Building safeguards and flexibility into a project

Safeguards may be built into projects, including insurance on various elements of the project (which increases the project cost but reduces risk to the project entity); providing for physical contingencies (really a form of self-insurance); adding a premium to the discount rate used in calculating the NPV of the project, or arbitrarily lowering the output values and/or increasing the input cost estimates in calculations of the ERR or the NPV.

These approaches may not be sensitive to the uncertainties identified. For example, adding a premium to the discount rate penalizes future costs and benefits more than present or early costs and benefits, and this is not necessarily related to where the main uncertainties exist. On the other hand, an arbitrary increase in costs (e.g., contingency or insurance) and/or decrease in benefits would, for any given discount rate, suggest that uncertainty concerning future values is less important than uncertainty concerning present or early values. This may not be in keeping with the levels and timing of uncertainty identified. Despite their shortcomings the approaches suggested are used widely as a convenient way to reduce the chance of failure or a lower than expected rate of return. It essentially amounts to the same thing as saying that the acceptance criterion is made more stringent, i.e., a project has to show better than marginally acceptable performance. Adding a contingency allowance for physical uncertainty is likely to be the preferable way to treat the problem, since it does not tend to hide what is being done from the decisionmaker.^[35]

Projects can be designed with specific contingencies in mind. For example, in the case of an industrial plantation project planned for Tanzania, it was recognized that a principle uncertainty facing the project would be that the yet-to-be built pulp and paper mill, which would use the wood, would not be built. Contingency plans for the project, in the unlikely event that the mill was not built, were (.1) gradually to scale down the planting program and stop it after five years, and (2) grow the trees planted to a 25-year sawlog rotation instead of the shorter planned pulpwood rotation. The project analysis showed that there would be an acceptable market for the resulting volumes of sawlogs. The same type of contingency planning was included in the Korean fuelwood plantation program, by planting a part of the area with dual purpose trees, i.e., ones that could be used both for fuel and timber.

Two additional points should be mentioned about uncertainty. First, uncertainty is often associated with the objectives for a project and the appropriate criteria for measuring the contribution of a project toward meeting objectives. This topic is not discussed here, mainly because it fits better in a discussion of sector planning, i.e., it is a question that transcends the subject of planning a particular project, given an objective. Objectives for a given project should be derived from a more general evaluation of the present condition of the sector and what goals it should be moving toward. The main problem with objectives at the project level relates to lack of definition. There is no sense in planning projects and project alternatives if objectives are not first defined explicitly. Criteria follow logically if objectives are defined. However, there are cases where criteria are poorly specified, mainly because objectives conflict or are loosely defined. The uncertainty in such cases is related to the lack of specified trade-off functions for the various conflicting objectives. Sensitivity analysis can contribute information on which decisionmakers can base subjective judgements regarding trade-offs. The uncertainty involved really relates to uncertainty concerning the relative values placed on various objectives by society or decisionmakers.

Second, a logical question is, How much should be spent on reducing uncertainty? In general, the amount spent depends on the nature of the project and the available budget. In some cases, slight additional effort/expenditure can result in a marked reduction in uncertainty. In other cases, substantial expenditure will have little impact on reduction of uncertainty. Judgement based on past experience and knowledge about information availability and cost of information will provide some idea of the particular cost/benefit relationship facing the analyst. As illustrated in box 7.4, how much reduction of uncertainty is worth to the decisionmaker is a judgmental question which has to be answered for each case separately.