Monthly Archives: May 2013

239 – Ranking environmental projects 5: Estimating and measuring values

Episode 5 in this series on principles to follow when ranking environmental projects. This one is about how to estimate the value or importance of environmental assets, and how to include that in the ranking process. 

We’ve seen that measuring the benefits of an environmental project requires attention to two aspects: the change in the physical condition of the environment, and the resulting change in the values generated by the environment (PD238). Suppose we have information about the change in physical conditions. How should we convert that to a measure of value or importance that we can use to rank projects? We need to do this in a way that is consistent between the different projects that we’ll want to compare.

Let’s consider three options, which are quite different in nature, but which are all actually used in real-world environmental programs.

(a) Scientific principles

Scientists often use rules of thumb (sometimes captured in an ‘Environmental Benefits Index’) to evaluate the relative importance of different potential environmental investments. An Australian example is the ‘habitat hectares’ concept, which is used by the state government in Victoria to evaluate proposed projects. A US example is the Environmental Benefits Index developed by the Natural Resources Research Institute (NRRI) of the University of Minnesota Duluth (http://beaver.nrri.umn.edu/EcolRank/). This consists of measures of soil quality risk, water quality risk and habitat quality, each scored out of 100, and then added up to give a total score out of 300.

Key strengths of this approach include:

  • The index is based on relatively sound knowledge of the natural systems.
  • Once the system has been developed, the approach is relatively efficient to apply to many potential projects.

But it also has some weaknesses:

  • The resulting Index scores reflect the values of experts, and there is plenty of evidence that experts and the general community sometimes think differently about what is important.
  • Environmental Benefits Indexes are set up to evaluate particular types of environmental benefits and cannot evaluate projects that generate different types of benefits. For example, the NRRI’s Index is no use for evaluating projects that protect threatened species or reduce air pollution. They can only rank projects of a reasonably similar type.
  • Often Environmental Benefits Indexes are not designed in a way that allows the required with-versus-without the project comparison. The NRRI index is an example. Even if we know what difference the project will make to environmental condition, this index would not help us value that difference. This could potentially be addressed by improving the design of the Index, although that would require considerable effort and resources.
  • Any system based on scoring, rather than dollars, cannot tell us whether the benefits of a project would exceed its costs. It can tell us how projects should be ranked, but not where the cut-off line should be for projects that are or are not worth funding. In most cases where projects are being ranked, this is not a serious problem because the overall budget is already determined. From a practical perspective, the relevant cut-off line is where the money runs out.

(b) Deliberative processes

‘A “deliberative process” is a process allowing a group of actors to receive and exchange information, to critically examine an issue, and to come to an agreement which will inform decision making’ (Gauvin 2009). It involves discussion, debate, and consideration of all information that is considered relevant. Multi-Criteria Analysis often employs this approach, although other approaches can use it as well.

Strengths:

  • There is scope to involve both experts and community members to ensure that both perspectives are considered.
  • The approach may be seen by stakeholders as being more transparent than the other approaches.
  • There is an opportunity for participating non-experts to receive detailed information and to participate in discussion and debate about the issues. This means that the outputs are likely to be better informed and better considered than is possible in survey-based approaches.
  • The approach is very flexible. All types of benefits and costs can be considered.
  • It is possible to generate a large number of valuations relatively efficiently – certainly more cheaply than conducting non-market valuation surveys for each project.

Weaknesses:

  • Participants may have vested interests or particular perspectives and may not reflect broader community interests or concerns.
  • While the flexibility of the approach is an advantage up to a point, the lack of theoretical rigour can be a problem, resulting in project rankings that don’t actually reflect the participants’ own values. In other words, too much flexibility can be a problem, particularly if the process goes beyond just looking at values. For example, when it comes to ranking projects, participants should not be free to choose to include costs in any way other than by dividing them into benefits (see PD236). Some things that people often choose to do in this space are just wrong (which is why I’m writing this series).
  • If the output is a score, rather than a dollar value, the approach cannot tell us whether the benefits of a project would exceed its costs.

(c) Dollar values

Environmental economists put a lot of effort into valuing environmental benefits in dollar terms, using a variety of techniques. (See PD218 to PD221 for details.)

Strengths:

  • Of the three approaches, this one is likely to best reflect broad community attitudes. It is more independent and less at risk of reflecting the preferences of vested interest groups.
  • It allows comparisons across completely different types of benefits.
  • It is more rigorous – less ad hoc than scoring-based approaches.
  • It allows us to determine whether the benefits of a project outweigh its costs.

Weaknesses:

  • Respondents to non-market valuation studies may know very little about the things they are being asked to value.
  • Conducting separate valuation studies for each project would be prohibitively expensive. Transferring benefit estimates from other similar projects can help to overcome this problem.
  • The survey-based methods have been criticised by some economists for relying on hypothetical questions and for giving results that don’t seem plausible in some cases. While this debate is interesting, in practice the quality of information from these surveys is probably higher than some other information we need to include in the process. For example, information about the cause-and-effect relationship between management and environmental conditions is often very weak indeed.

Which is best?

Some people are quite definite in their preferences for one or another of these approaches, or particularly dislike one of them. In my view, it’s not a clear-cut decision. They each have pros and cons, and one’s choice of which to use may vary depending on the circumstances. The weaknesses that concern me most are: the inability of many Environmental Benefits Indexes to compare outcomes with and without the project; the excessive flexibility of some deliberative approaches, giving participants the flexibility to do dumb things; and the expense of doing comprehensive valuation surveys.

wellstead_estuary2My advice is to weigh up the pros and cons and use whichever approach makes most sense for a particular program. My caution would be that this advice applies specifically to the part of the process that estimates values. For the other parts of the process, and for decisions about how to combine the various bits of information to inform decisions, see the other posts in this series.

A practical compromise

In developing INFFER (Pannell et al. 2012) we attempted to create an approach to valuation that draws on the combined strengths of the three approaches outlined above, while limiting their weaknesses. The approach we developed:

  • Can use scientific information if it is available
  • Recognises that the relevant benefit is a difference (with minus without the project)
  • Can be elicited in a deliberative process involving both experts and community members
  • Can be cheap and quick enough to be practical in cases where there are limited resources for estimating values, or where many valuations are needed in a short time
  • Provides dollar values
  • Can use results from non-market valuation surveys if available

Here is how it works. Define P’ as the physical condition of the environmental asset in good condition. For example, it could be an environmental condition of 100 in Figures 4 and 5 (PD238).

Now V(P’) is the value of the environmental asset at condition P’. It includes all the different types of values (market and non-market) that are relevant to this environmental asset. In Figures 4 and 5, if P’ = 100, V(P’) would be $1 million.

Finally, define W as the difference in values between P1 (physical condition with the project) and P0,(physical condition without the project) as a proportion of V(P’).

pd239e1

Then we measure the project benefit as V(P’) × W:

pd239e2b

pd239e3

pd239e4

So V(P’) × W is equivalent to the correct measure of benefits, V(P1) – V(P0) (as outlined in PD238).

The benefit of re-organising the benefits into V(P’) and W is that, in my experience, it helps people think clearly and ask the right questions in a situation where they are not going to conduct a non-market valuation survey. V(P’) sets an upper bound for the benefits of the project – obviously, the value of the project can’t be more than the value of the environmental asset in good condition.

In INFFER, we ask users to score V(P’) relative to a set of examples – a table of well-known environmental assets with suggested V(P’) scores. We define V(P’) as being worth $20 million per point. This is often done in a group discussion environment, involving a variety of stakeholders.

A risk with this (and other deliberative processes) is that people may provide values that are too high (e.g. see PD213). A process of reviewing assumptions and comparing them across projects is needed to reduce this risk.

Defining W as a proportion of V(P’) helps to highlight that the benefits of the project must be proportional to the effectiveness of the project, which is often missed when people develop their metric for ranking projects.

For example, suppose there are two alternative projects for Asset A. Project (i) would increase the asset value by a factor of 0.3 and Project (ii) would increase it by 0.6. If everything else is equal, Project (ii) would generate benefits that are twice as large as those from Project (i). The metric has to reflect that. This is achieved by multiplying by W.

Finally, a mistake I’ve seen is to exclude any measure of values from the ranking process. One senior bureaucrat told me that she was opposed to including them because of the risk of them generating controversy. At other times, people seem to simply overlook them. The consequence of this is that the organisation will tend to bias its funding towards less valuable projects. There is an increased risk that they will incorrectly rank projects addressing less-valuable assets relative to more-valuable assets.

Further reading

Gauvin, F.-P. (2009). What is a Deliberative Process? National Collaborating Centre for Healthy Public Policy, Quebec, http://www.ncchpp.ca/docs/DeliberativeDoc1_EN_pdf.pdf

Pannell, D.J., Roberts, A.M., Park, G., Alexander, J., Curatolo, A. and Marsh, S. (2012). Integrated assessment of public investment in land-use change to protect environmental assets in Australia, Land Use Policy 29(2): 377-387. Journal web site here ♦ IDEAS page for this paper

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Designing a practical and rigorous framework for comprehensive evaluation and prioritisation of environmental projects, Wildlife Research 40(2), 126-133. Journal web page ♦ Pre-publication version at IDEAS

238 – Ranking environmental projects 4: Environmental condition and values

Episode 4 in this series on principles to follow when ranking environmental projects. This one discusses how to think about environmental conditions and the resulting environmental values. 

In the previous post I said that the benefit from an environmental project is the difference between the environmental values with the project and without the project. This time I will break that down a bit. The point of this post is that there are two parts to that change in environmental values: a change in the physical condition of the environment, and a resulting change in the values generated by the environment (in other words, the value of the environmental services).

inletSo, to estimate the benefits of an environmental project, you need to (a) predict the physical environmental conditions with and without the project, and (b) translate the difference into a measure of value or importance or significance.

This raises the question, what is the relationship between the physical condition of an environmental asset and the values it provides to the community? As environmental conditions improved, values would increase, but is it a simple linear increase, or something else? To some extent, this would depend on how you measure the condition of the environment, but a common result in the environmental economics literature is for values to increase at a decreasing rate, as illustrated in Figure 4. (In fact, we see this sort of relationship for all sorts of things, not just environmental goods.)

pd0238f1

Figure 4.

 

The relationship in Figure 4 relates to the values that people in the community put on environmental improvements, but it is also consistent with one aspect of the ways that environmental scientists tend to think about values: if something is rarer, each unit of it is considered more valuable.

In theory, if you could quantify environmental conditions and knew the relationship between condition and value, you could read off the change in value from this graph. For example, Figure 5 shows that a project that increases the environmental condition score from 40 to 60 results in an increase in value from about 0.8 to 0.9. If we are measuring the value in millions of dollars, the benefit of that project would be $100,000.

Benefit = V(P1) – V(P0) = V(60) – V(40) = 0.9 – 0.8 = 0.1 $million = $100,000

where V is value, which depends on the physical condition, P1 is physical condition with the project and P0 is physical condition without the project.

pd0238f2

Figure 5.

 

In practice, we may or may not have a system for quantitatively scoring the type of environmental condition we are interested in in a particular case, but we should at least be able to describe the environmental conditions in words, with and without the project. Then we have to translate those into a measure of value (the topic of the next post).

Amongst the systems I’ve seen in use for ranking environmental projects, a surprising number make no attempt to evaluate the difference that the project will make to environmental conditions. Without that, there is no prospect of obtaining a meaningful estimate of the benefits from the project, so decision making (and ultimately the environment) suffers.

Further reading

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Designing a practical and rigorous framework for comprehensive evaluation and prioritisation of environmental projects, Wildlife Research 40(2), 126-133. Journal web page ♦ Pre-publication version at IDEAS

237 – Ranking environmental projects 3: With vs without

Episode 3 in this series on principles to follow when ranking environmental projects. This one discusses the “with versus without” principle for estimating the project benefits. 

Through the series, we will cover a number of points about the estimation of benefits from an environmental project. Initially, to keep things simple, I’ll talk about the case where there is a single type of benefit being generated by an environmental project (e.g. a threatened species is being made safer). In later posts I’ll talk about cases with multiple types of benefits from the same project.

This first point is deceptively simple. It is that the benefit of an environmental project is the change in value of the environmental asset as a result of the project. In other words, it is a difference: the difference between the environmental value with the project and without the project.

So, to estimate the benefits of a project, you need two pieces of information: the environmental values with the project and the values without the project. Usually, when we are evaluating a project, the project has not yet been implemented. In that case, both of the required pieces of information have to be predicted. You can’t observe them, because they are in the future.

Note that comparing environmental values “with versus without” the project is not the same as comparing values “before versus after” the project. The reason is that the condition of the asset would probably not be static in the absence of the project. For example, it may be that the asset would degrade in the absence of the project, but its condition would be improved by the project (relative to its current condition). This is illustrated in Figure 1.

The graph illustrates a case where the asset currently has a value of 57 [labelled (1)]. (The 57 is just some measure of value – we’ll discuss values more in later posts.) Without the proposed project, the value is expected to decline steadily, to a score of 37 after 25 years [labelled (3)]. With the project, value would increase to a score of 76 after 25 years [labelled (2)].

pd0237f1c

Figure 1.

 

Clearly, in this example, the benefits of the project grow over time (the two lines diverge in Figure 1). Ideally, we would estimate the benefits in each year after the project is implemented and add them up (after allowing for discounting, which we’ll cover in a later post). A practical simplification is to estimate the environmental benefits based on the difference in the asset value with and without the project in a particular future year. For example, we might choose to focus on 25 years in the future, and estimate values at that date with and without the project. In doing this, we need to be careful that we deal appropriately with time (see a later post for details).

creswick_bridgeAssuming we go with that simplified approach (focusing on benefits at year 25), the relevant measure of project benefits for ranking projects is (2) minus (3). I have seen ranking systems which use (1) minus (3), (2) minus (1), (1) alone or (2) alone, and sometimes more than one of these in the same ranking system, but they are all irrelevant. If you include (2) minus (3) you should not include any of the others listed. To do so will just make the rankings worse.

Because of the “with versus without” principle, a project can generate benefits even if it does not completely prevent degradation of the environmental asset. As long as it slows or reduces degradation, this should be measured as a benefit. Figure 2 shows an illustration of this. In this example, future asset condition with the project (2) is below the initial asset condition (1), but is above future asset condition without the project (3). Since the project benefit is (2) minus (3), the benefit is positive.

pd0237f2c

Figure 2.

 

On the other hand, a project that superficially appears to generate large benefits may actually not do so, because those benefits would have been generated even without the project. In other words, the benefits are not ‘additional’ to what would have happened anyway. The without-project line in the graph would be almost as high as the with-project line, so the difference between them (= the benefit of the project) would be minimal (Figure 3).

pd0237f3c

Figure 3.

 

For example, suppose that a proposed project encourages farmers to adopt a new type of environmentally beneficial crop, where that crop is much more profitable than farmers’ existing crops. If the private benefits are large enough, it’s a safe bet that the farmers would have adopted the new crop even without the project. It would have been promoted by word of mouth and by private farm business consultants. Adoption of the crop for commercial reasons would have generated environmental benefits as a spin-off.

Making good predictions about the “without project” scenario can be quite difficult, requiring good knowledge of the environment, the relevant management practices and the people whose behaviour matters. Weak thinking about the “without” scenario for environmental projects is a common failing, sometimes leading to exaggerated estimates of the benefits.

Further reading

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Designing a practical and rigorous framework for comprehensive evaluation and prioritisation of environmental projects, Wildlife Research 40(2), 126-133. Journal web page ♦ Pre-publication version at IDEAS

236 – Ranking environmental projects 2: Divide by costs

Episode 2 in this series on principles to follow when ranking environmental projects. This one discusses one aspect of the metric used to rank projects: how to include costs.

Suppose you manage an environmental program that has a budget available for spending on environmental projects and there is not enough money to fund every proposed project. You have to decide which projects to fund. How should you do it?

The first principle is that projects should be ranked using a metric (a formula) that consists of a measure of project benefits divided by a measure of project costs. Economists call this metric a Benefit: Cost Ratio (BCR).

pd236e1

There are plenty of project ranking metrics out there in actual use that don’t do this. Some subtract costs instead of dividing them, and some (remarkably) ignore costs entirely. These are mistakes that are costly to the environment.

To illustrate, consider the following three hypothetical projects, with the indicated benefits (B) and costs (C). Because the budget is limited, the first project we should choose is the one with the highest benefits per unit cost (the highest BCR) = project 1. But if we rank according to BC the top ranked project seems to be project 2, while ranking according to B (ignoring costs) tells us that project 3 is best.

ProjectBCBCRB - CRank(BCR)Rank(B - C)Rank(B)
15154123
2723.55212
3871.11331

The loss of environmental values from using the wrong metric (i.e., ranking according to BC or B) depends on how tight the budget is. Assuming that the budget is enough to fund 10% of projects, the loss of environmental benefits is 12% for B – C and 19% for B (based on simulating 1000 funding rounds with 100 potential projects in each).

richardson_riverIn other words, fixing up the formula is like increasing the program budget by 14% or 23%. It’s much easier to fix the formula than to increase the budget!

In the examples above, I’ve assumed that we know what the benefits and costs would be for each project. Later posts in this series will deal in detail with how we should estimate the benefits and costs. For now I’ll just make these two observations.

The benefits used in the ranking metric should be the benefits of the proposed intervention or project, not the total benefits of the environmental asset. What difference can be made by the intervention or project, and how important is that difference?

The costs should also represent the costs of the intervention or project. If this project did not go ahead, what level of resources could be diverted to other uses?

p.s. (9 May 2013). A slightly more technical issue: it is sometimes claimed that BCRs are flawed because they can be manipulated by transferring costs from the denominator to the numerator. For example, suppose that a proposed project has benefits of $10m, program costs of $2m (requested from the funding program) and other costs of $1m (from other sources, such as the private sector). We could potentially calculate the BCR as 10/(2+1) = 3.3, or else as (10-1)/2 = 4.5. However, there is no ambiguity about the correct way to do this: what should go into the denominator are the costs that are in limited supply from the perspective of the decision makers in the funding program. They are trying to choose the projects that generate the most net benefits per dollar that they have to allocate. So if the other costs are not drawn from a fixed budget, the correct procedure is to subtract the other costs from the benefits, meaning that the correct BCR for the above project would be 4.5.

Things get a bit tricky, however, if projects also require ongoing maintenance funding beyond the current project, and the budget for maintenance funding is expected to be fully allocated. This is realistic for many (probably most) projects. In this case, there are actually two constraints that must be satisfied: the current program budget and the long-term maintenance budget. Strictly, in this situation, projects cannot be ranked using a single formula as a metric. The program would need a mathematical programming model to select which projects deliver the most benefits while satisfying both constraints. In practice, after testing various approaches, I believe that a reasonable approximation is to add up both costs (short-term program costs and long-term maintenance costs) and include the total as the denominator in the single formula.

Further reading

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Designing a practical and rigorous framework for comprehensive evaluation and prioritisation of environmental projects, Wildlife Research 40(2), 126-133. Journal web page ♦ Pre-publication version at IDEAS

235 – Ranking environmental projects 1

Environmental organisations need to rank projects that they could potentially invest in. Often it is done poorly. This post starts a series on how to do it better.

The funding available for environmental projects and policies is a small percentage of the money we would need to deal comprehensively with all environmental problems. As a result, whether we like it or not, we have to choose what we do and don’t protect. Even programs that don’t explicitly prioritise their environmental investments do so implicitly – they just do it in a non-transparent, and usually very poor, way.

In my experience, the difference in potential environmental outcomes between poor prioritisation processes and good ones is enormous.

Doing a good job of ranking the investment options is not that hard if you are aware of a few principles, but it seems to me that most people who are responsible for deciding how environmental funds get allocated are not aware of these principles. Indeed, some of the most commonly used approaches to ranking environmental projects are guaranteed to result in very poor rankings. As a result, we miss easy opportunities to deliver much greater environmental outcomes.

My aim in this series of posts is to outline a set of relevant principles and insights that will help environmental decision makers choose the best projects. My focus is on collecting and analysing the information needed to provide high-quality project rankings. There is another set of issues about how the rules of the program are designed to provide incentives for its participants to behave appropriately (e.g. Pannell and Roberts 2010), but I won’t be covering those here. I’ll be talking about information, calculations and clear thinking – stuff that is easy to get right if you know what you are doing.

My aim is to help with practical decision making. As a result, I’ll be talking about the possibility of cutting corners by simplifying aspects of the process. You’ll see that I’m not averse to well-considered simplifications, but very wary of the risk that some simplifications will sabotage the whole process. For a practical system, simplifications are essential, but bad simplifications are disastrous.

Throughout, I will be assuming that the aim is to provide the most valuable environmental outcomes for the available resources.

What is being ranked?

The first requirement is to be clear about what is being ranked. Sometimes programs set out to rank a set of projects that they might invest in. The projects should define what would be done, to which environmental assets, where, and by whom.

At other times, programs seek to rank a set of environmental assets, with no explicit project activities defined. (I’ll use the term “environmental asset” to refer to any identifiable feature, entity, place, or species that might become a target for investment.) There is a risk here – if you don’t define the project activities for an environmental asset, you cannot rank them on the basis of providing the most valuable outcomes.

The problem is that the environmental value for money depends on the answers to questions like, “what is the technical feasibility of protecting the asset?”, “to what extent would the community cooperate?” and “what would it cost to protect the asset?” However, those questions can only be answered for a particular set of actions or interventions.

To further illustrate the point, various different projects could be defined for the same environmental asset. One potential project might have very ambitious goals, aiming to return the asset to pristine condition, while another might aim for a moderate improvement in its condition. Some of these different projects for the same asset may offer relatively good value for money while others don’t (e.g. Roberts et al. 2012). So you cannot conclude that investing in any particular asset is good or bad without being clear about the project actions that will be undertaken.

If the analysis is limited to environmental assets, not projects, then it is important to be aware of what can and cannot be done with the results. What you can reasonably do is filter the assets to identify ones where it is relatively likely that a well-designed project would deliver worthwhile benefits. This could be done using variables such as:

  • the value or significance of the assets,
  • the levels of degradation they have already suffered or are likely to suffer in future, and
  • the feasibility of managing them (in a loose general sense that doesn’t require specification of particular management actions).

You should not be making final decisions about which assets received funding, because that does require the specification of projects. Rather, you would be concluding that some assets are probably not worth considering further, and so not worth developing projects for.

Even this is not without risks. Because you are not looking at all of the relevant information, there is a chance of excluding some assets that would actually be worth investing in. For example, you might exclude investment in a particular asset because it seems likely to provide only modest benefits, but if the cost of the project is low enough, it could still be worth doing. With this process of filtering assets, you would miss out on cases like that.

However, it still might be worth filtering assets as part of a more comprehensive process. Indeed that is exactly what we do in Step 1 of INFFER (the Investment Framework for Environmental Resources) (Pannell et al. 2012). This is a simplification we use to reduce the cost of the system. If we can knock out some potential investments based on partial information, it takes less work to properly evaluate and rank a reduced set of potential projects.

If you must make final investment decisions based on assets, not projects, you need to imagine a notional project for each asset. Even a rough-and-ready notional project definition would be better than nothing.

Further reading

Pannell, D.J. and Roberts, A.M. (2010). The National Action Plan for Salinity and Water Quality: A retrospective assessment, Australian Journal of Agricultural and Resource Economics54(4): 437-456. Journal web site here ♦ IDEAS page for this paper

Pannell, D.J., Roberts, A.M., Park, G., Alexander, J., Curatolo, A. and Marsh, S. (2012). Integrated assessment of public investment in land-use change to protect environmental assets in Australia, Land Use Policy 29(2): 377-387. Journal web site here ♦ IDEAS page for this paper

Roberts, A.M. Pannell, D.J. Doole, G. and Vigiak, O. (2012). Agricultural land management strategies to reduce phosphorus loads in the Gippsland Lakes, Australia, Agricultural Systems 106(1), 11-22. Journal web site here ♦ IDEAS page for this paper