Monthly Archives: June 2006

81 – The cult of the asterisk

Researchers in many fields have an essential reliance on statistics, but researchers often apply them mechanically, and sometimes they misrepresent what the results really mean. In particular, statistical “significance”, while a useful concept, can be a poor indicator of the importance of a variable in an economic or management sense.

Statistical analysis is a standard and essential tool of researchers in most scientific disciplines. Statistical methods can be things of beauty and power. They allow us to make rigorous statements about the probabilities of certain ideas being true, based on the evidence embedded within a set of data. Unfortunately, statistics are often applied in a mechanical way, and this can lead to problems.

For example, in the early days of statistics, someone decided that it would be reasonable to choose 5% as the cut off point for uncertainty about the idea being tested. If the statistics showed that there was less than 5% probability of being in error when concluding, for example, that there was a positive relationship between fertilizer input and crop yield, then we would accept that there probably is a relationship. If the probability of being in error was more than 5%, we would conclude that the idea was not true. (Strictly speaking, we would not reject the idea that it was not true, although in practice, this usually is taken as evidence that it is not true.)

Of course, the 5% cut-off is just an arbitrary choice. Why not 10%, or 1%, or 3.3%? Recognising this arbitrariness, researchers often put asterisks next to their statistical results to indicate just how low the cut-off can be set and still conclude that the result is “significant”: e.g. one asterisk for 10%, two asterisks for 5%, three asterisks for 1%. The more asterisks, the better.

While that strategy avoids some of the arbitrariness in the general approach, it doesn’t get away from another problem: that this approach to testing the truth of an idea is unbalanced in the way it deals with different sorts of potential errors.

To illustrate, return to the example of testing for a relationship between fertilizer input and crop yield. In the standard approach to statistics, we start by assuming that there is no relationship (that the slope of the relationship is zero) and test whether this appears wrong. A zero slope is established as the point of comparison.

[From this point, the way that standard statistics proceeds can be a bit hard to get your mind around. I’ll warn you that the next paragraph might be a bit of a brain twister. I can’t make it any simpler, because it is trying to represent the way statistics actually operates.]

We then ask ourselves the following: assuming that the slope actually is zero, what is the probability that a non-zero slope as big as the one we observe in the data set would occur just by chance, as a result of random fluctuations. The bigger the observed slope, the less likely it is that it could have occurred just by chance, and therefore, the more likely it is that the slope really is non-zero. If the probability of getting the observed slope by sheer chance is less than 5%, we reject the starting assumption that the slope is zero.

This implies that, if we looked at lots of examples where the slope really was zero, we would mistakenly reject the idea of a zero slope 5% of the time (and we’d correctly accept that there is a zero slope the other 95% of the time). Clearly, this approach is conservative in avoiding the error of concluding that there is a slope when there isn’t one. (This is the so-called Type-I error that we are taught in statistics.)

On the other hand, if there actually is a positive slope, the approach has a tendency to lead you to a conclusion that there isn’t one (a Type-II error). If the slope isn’t big enough, we conclude that there is no slope, rather than concluding that there is a low slope. There is, in a sense, a bias towards accepting that there is no slope.

The approach effectively gives a high weight to avoiding Type-I errors, but pays little or no attention to Type-II errors. But who’s to say that Type-I errors are much more important than Type-II. In reality, Type-II could easily be more important in an economic sense.

A related problem is that, just because a variable is statistically significant (at 5% or any other level), it does not necessarily follow that the variable is important, in the sense of having a major influence on the issue. Even if variable X has little effect on variable Y, its effect might be statistically significant, if the relationship is very tight, meaning that there is little random scatter in the data, or if the data set is large enough. Statistical significance indicates that the relationship is real, not that it is important.

Various writers have pointed this out. For example, Dillon (1977) puts it beautifully:

“[Through] tests of statistical significance (the “cult of the asterisk”) involving mechanical application of arbitrary probabilities of accepting a false hypothesis, traditional procedures … have aimed at protecting the researcher from “scientific error”. In doing so, these procedures have led to a far greater error of research-resource waste. The farmer’s problem is not whether or not there is a 5 per cent or less chance that a crop-fertilizer response function exists. His problem is how much fertilizer to use. Even if the estimated function is only statistically significant at the 50 per cent level, it may still be exceedingly profitable … for the farmer to base his decisions on the estimated function.”

“In the mechanical fashion in which they are usually applied, significance levels have no economic relevance, except by chance, to farmer decisions about best operating conditions. (Dillon, 1977, p. 164).”

He was referring to the use of statistics in the analysis of agricultural experiments. Unfortunately, the problem is just as serious in economics. McCloskey and Ziliak (1996) went through all of the statistical papers published in the American Economic Review during the 1980s to check how many researchers were relying solely on statistical significance as their measure of real-world importance.

The answer was, most of them. Even in what is arguably the highest prestige economics journal, only about 30% of articles considered more than statistical significance as being decisive in drawing conclusions about the real world, or made any distinction between statistical significance and substantive importance.

That was in the 1980s, but little has changed. The cult of the asterisk is alive and well. I find it particularly disappointing that it affects economics so deeply. One would have thought that, given their disciplinary interests in decision making, economists would have known better.

Traditional statistics is an important tool, but it can be useful to supplement tests of statistical significance by also calculating other indicators of the importance of the variables. For example, in Abadi et al. (2005) we used “importance” indicators, representing how much difference the variables make to the predicted results. Essentially, our importance indicators answered the following question: if we varied a variable over the range that is present in the data, how much difference does it make to the model’s output? Predictably, we found that not all statistically significant variables were important, and not all of the important variables were statistically significant.

David Pannell, The University of Western Australia

Further Reading

Abadi Ghadim, A.K., Pannell, D.J. and Burton, M.P. (2005). Risk, uncertainty and learning in adoption of a crop innovation, Agricultural Economics 33: 1-9.

Dillon, J.L. (1977). The Analysis of Response in Crop and Livestock Production, Pergamon, Oxford.

McCloskey, D. N. and Ziliak, S. T. (1996), “The standard error of regressions”, Journal of Economic Literature 34, 97-114.

80 – Public benefits, private benefits: the final framework

This is the seventh and final instalment of a series that examines a simple framework for choosing environmental policy instruments, as outlined in PD#73. The framework is based on levels of public and private net benefits of changing land management, and a set of simple rules. This time we pull together refinements developed for each part of the framework over the past five Pannell Discussions, and present a revised version of the overall framework.

In PD#73 I showed how a set of simple and reasonable rules can lead to a useful map of efficient policy instruments. The context is an environmental manager considering prospective projects to change land use in particular ways on particular pieces of private land. The map shows that the choice of instruments depends crucially on the levels of public and private net benefits from those projects. A particular project to change land use in particular ways on particular pieces of land would be represented by a dot somewhere on Figure 1. Depending on where the various dots lie, different types of policy response are recommended.

In the past five Pannell Discussions we have looked in more detail at the individual areas of the map, and refined its recommendations. This article pulls together those refinements to present a revised overall framework.

The refined map shown in Figure 1 is based on environmental managers requiring a benefit:cost ratio (BCR) of at least 1.0 in order to invest in incentives or extension.

Figure 1. Efficient policy mechanisms for encouraging land use on private land, refined according to PD#75, PD#76 and PD#78, assuming managers require BCR > 1.

In broad terms, the framework indicates the use of:

  • positive incentives if the public net benefits of land-use change are high, and the private net benefits are not too negative;
  • extension if the public net benefits of land-use change are high, and the private net benefits are moderate;
  • no action if private net benefits are positive and public net benefits are not sufficiently high;
  • no action if private net benefits are greater than public net costs;
  • negative incentives if private net benefits are less than public net costs;
  • no action if public net benefits and private net benefits are both negative;
  • technology development if private net benefits are negative and public net benefits are not sufficiently high to warrant incentives;

Figure 1 is broadly similar to the original map in PD#73, with the main difference being in the extension area, which is more targeted to projects with higher public net benefits or lower but still positive private net benefits.

Figure 2 shows a comparable diagram based on a required BCR of at least 2.0, which is probably a more reasonable guide to investment than Figure 1, given that program resources are limited and there are more worthwhile projects available than the program can afford to fund. (Also, we might need a BCR of at least 2 to outweigh the overhead costs of running the program.) This more targeted strategy shows that, broadly speaking, the higher priority projects are those where private net benefits are closer to zero, and/or public net benefits are more extremely positive or negative.

Figure 2. Efficient policy mechanisms for encouraging land use on private land, refined according to PD#75, PD#76 and PD#78, assuming managers require BCR > 2.

A much smaller number of projects would qualify for incentives or extension in the more targeted approach of Figure 2. For example, over 35% of the area of Figure 1 is occupied by incentives or extension, whereas in Figure 2, they occupy less than 15%. If we allow for the reality that most projects involve negative private net benefits, the proportion qualifying as high-priority targets for intervention is lower again.

As noted in PD#78, most agricultural land probably falls into the technology development area. For most land, the best available environmental projects involve negative private net benefits and positive, but not extremely high, public net benefits. This highlights the important role of technology development. It has been relatively neglected in current programs.

Finally some general observations about the framework. The recommendations in Figures 1 and 2 (like all of the recommendations in the framework) depend on the landholders having reasonably accurate perceptions about the private net benefits of adoption. If this is not true, there may be roles for extension, positive incentives or negative incentives in other parts of Figures 1 and 2.

It is notable that the choice of policy response depends at least as much on the level of private net benefits from the land-use change as on the public net benefits. Indeed, in the more targeted version in Figure 2, results are even more sensitive to private than to public net benefits. This is an important finding as many environmental managers focus predominantly on the public benefits, but pay little attention to the estimation of private net benefits. As a consequence, they are under-informed about the landholders’ likely responses to any proposed changes in land use, which is one of the key factors that should influence the choice of policy response.

This begs the question, how should environmental managers estimate the costs and benefits? A glib answer is, “as best they can”. In the case of public net benefits, the framework does not require environmental managers to do things that they should not already be doing. Somehow they are choosing which environmental projects are of highest priority, so there must be some assessment of the environmental benefits, even if only implicitly. It is unrealistic to expect that projects could be ranked according to their environmental benefits with any great precision, but even relatively qualitative ratings could be applied within this framework.

To estimate private net benefits, one option is to invest in some good quality economic modelling. Another is to look at what farmers are currently doing. If they are choosing not to adopt a practice that has been around for a while and with which they are familiar, this provides a strong indication of their assessment of its private net benefits (including issues beyond just short-term financial returns). A third option is to run a conservation auction, in which landholders reveal their willingness to act in response to a subsidy level chosen by them.

It is important to recognise that both categories of net benefits depend on several elements. The public net benefits are not simply the value of the environmental assets involved, and the private net benefits are not simply the profits from the new land use. Indeed, the private net benefits of a project (i.e. a specific set of land-use changes) would depend on:

  • the financial returns from the new land uses;
  • the financial returns from the land uses that are replaced (the “opportunity costs”);
  • any change in risks faced as a result of the change;
  • indirect impacts on other aspects of the farm system or on the farmer’s lifestyle;
  • the farmer’s own interest in the environmental outcomes.

The public net benefits would depend on:

  • the value or importance of the environmental assets that are affected by the changes;
  • the degree of degradation that the assets were facing or had already suffered;
  • the extent to which that degradation can be prevented or alleviated by the changes;
  • any lags in the response of the biological or physical system to the land-use changes.

Overall, the framework highlights the importance of targeting funds in environmental programs to selected areas, based on the levels of public and private net benefits. Currently, environmental managers do pay some attention to the level of public benefits when selecting their investments, but in my experience few pay adequate attention to the level of private net benefits, which, perhaps surprisingly, turns out to be even more important as a driver of policy decisions.

David Pannell, The University of Western Australia

Further Reading

A consolidated paper on the public benefits, private benefits framework (combines all seven related Pannell Discussions): Public benefits, private benefits, and the choice of policy tool for land-use change

79 – Public benefits, private benefits, and technology development

This is the sixth instalment of a series that examines a simple framework for choosing environmental policy instruments, as outlined in PD#73. The framework is based on levels of public and private net benefits of changing land management, and a set of simple rules. This time we focus on the use of technology development.

In PD#73 I showed how a set of simple and reasonable rules can lead to a useful map of efficient policy instruments. The context is an environmental manager considering prospective projects to change land use in particular ways on particular pieces of private land. In the past four Pannell Discussions we have looked in more detail at the individual policy mechanism choices, and refined the map that was presented in PD#73. This article deals with the last of the potential mechanisms: technology development.

In this context, technology development means development of improved land management options, such as through strategic R&D, participatory R&D with landholders, and perhaps provision of infrastructure. Examples could include plant breeding and selection to generate more productive perennial plants, or farming systems research to test and improve management of new plants in an agricultural context.

Figure 1 shows the revised map of efficient policy mechanisms from PD#76. The choice of mechanisms depends on the levels of public and private net benefits from the proposed changes in land management. A particular project to change land use in particular ways on particular pieces of land would be represented by a dot somewhere on Figure 1. Depending on where the various dots lie, different types of policy response are recommended. This section of the map is for cases where public net benefits are positive. It allows for adoption lags and learning costs (see PD#75).

Figure 1. Revised map of efficient policy mechanisms allowing for adoption lags and learning costs.

The reasons for recommending technology development in the lower-left area of the figure are: (a) Private net costs exceed public net benefits, so the management changes required in the project do not yield positive net benefits overall. This means that positive incentives are not appropriate; and (b) Technology development may provide a cost-effective strategy to generate new technologies or may alter existing technologies so that they do yield net benefits overall.

To explain what I mean in (b), here is an example. Suppose that point a Figure 1 represents a project that aims to plant 40% of the area of a watershed to woody perennial plants. The project would generate modest public net benefits per ha (about $15/ha/year in the figure), but it would do so at a net cost of $25/ha/year to private landholders. Without some incentive mechanism, the land-use change would not be voluntarily adopted, and there is no case to introduce an incentive mechanism because, overall, costs would outweigh benefits.

Technology development could be undertaken to develop new woody perennial plants that would be more profitable to farmers than their current land use in the relevant location, while providing at least the same public net benefits. If this technology development was successful, it would move the project to, say, point d. Although the recommendation for the new project would then be “no action”, the profitability of the new woody perennial plants would prompt voluntary adoption of the new plants, probably after a lag during which landholders learned about the technology and gained confidence in it. In this way, the investment in technology development would yield public environmental benefits by enhancing adoption via the mechanism of generating increased profits for landholders.

Alternatively, technology development could attempt to improve the environmental benefits of the technology, moving the project up to, say, point b. For example, this might involve developing new harvesting systems so that the woody perennials provided better habitat for wildlife. If the improvement was substantial enough, it would then be worth using positive incentives to get landholders to adopt the new management option. At point b, the incentives required to compensate growers for adopting would be less than the public net benefits from the land-use change itself.

If both dimensions could be improved, we might move the project to, say, point c, where extension is recommended, as private net benefits are positive enough for incentives to be unnecessary – landholders would adopt the change reasonably rapid without them.

The merits of technology development as a strategy for the indicated set of projects depends on a set of additional factors that cannot be illustrated on this diagram, including: the likelihood of R&D delivering sufficiently improved technologies, the time lag until delivery of improved technologies, and the cost of the R&D. Overall, R&D does have an outstanding track record of delivering improved technologies for agriculture, and in my view, it has been neglected as a strategy for investment in environmental programs. A particular attraction of technology development is its potential to prompt adoption of changed practices over large areas, without the need for incentives.

Figure 1 is perhaps a little misleading in that the role of technology development would not be limited to the indicated area; it could be an option in any part of the diagram, depending on the opportunities and the costs. It is particularly indicated as the recommendation for the lower-left area of Figure 1 because the more direct instruments (positive incentives and extension) are not suitable for that area, and yet much (probably most) agricultural land falls within this category, if we are thinking about some of the main resource degradation problems. Also, given the past neglect of technology development as a strategy, there are likely to be many unexploited opportunities for cost-effective investment in that area.

David Pannell, The University of Western Australia

78 – Public benefits, private benefits, and negative incentives

This is the fifth instalment of a series that examines a simple framework for choosing environmental policy instruments, as outlined in PD#73. The framework is based on levels of public and private net benefits of changing land management, and a set of simple rules. This time we focus on the use of negative incentives.

In PD#73 I showed how a set of simple and reasonable rules can lead to a map of efficient policy instruments (Figure 1). The context is an environmental manager considering prospective projects to change land use in particular ways on particular pieces of private land. The map shows that the choice of instruments depends crucially on the levels of public and private net benefits from those projects. A particular project to change land use in particular ways on particular pieces of land would be represented by a dot somewhere on Figure 1. Depending on where the various dots lie, different types of policy response are recommended.

In PD#74 and PD#75 we focused on the use of positive incentives and in PD#76 we looked at extension. This time we examine negative incentives more closely.

Figure 1. Efficient policy mechanisms for encouraging land use on private land.

“Negative incentives” means that landholders are encouraged to NOT change their land management in particular ways (e.g. clearing of environmentally valuable vegetation) using tools such as command-and-control regulation, environmental taxes, or, potentially, subsidies as a reward for not changing.

The reasons for recommending negative incentives in the bottom-right triangular area of Figure 1 are (a) projects in that area generate negative public net benefits, (b) they generate positive private net benefits, so landholders are likely to adopt the changed land-use practices unless they are prevented from doing so, and (c) the public net costs outweigh the private net benefits, so there are overall benefits to be gained from preventing the land-use changes that the private landholders would like to adopt.

In previous PDs we have seen that it is possible to refine the set of projects suited to a particular policy mechanism by accounting for further complexities, such as the likely lag to adoption, and the same is true in this case. In PD#75 we presented an illustrative (plausible) relationship between the adoption lag and the level of private net benefits from adoption. The lag to adoption would probably be low if private net benefits are high. As private net benefits fall, the lag (in the absence of incentives) would increase, and it would probably be very long indeed as the benefits of adoption are reduced to zero.

Allowing for this relationship, and assuming that monitoring and enforcement costs the same per ha as assumed for extension (PD#76), the region where negative incentives would pay off shrinks slightly, as shown by the BCR = 1 line in Figure 2. (BCR stands for Benefit:Cost Ratio – if BCR > 1, benefits exceed costs.) The shrinking occurs because, if there is a lag to adoption, the public outcomes are not as significant as they would have been without the lag – other projects with more immediate impacts would tend to rise in the priority list.

Figure 2. Benefit:cost ratios from use of negative incentives, allowing for adoption lags

In almost a mirror image to the result for positive incentives, if we seek to apply negative incentives only in cases where the BCR is higher, it will be in cases where the public net costs are higher and/or the private net benefits are low (see the BCR = 2 and BCR = 3 lines in Figure 2).

The recommendation of “No action” in the right-most triangle of Figure 1 implicitly assumes that we are happy with the current distribution of property rights, whatever that distribution is. For example, if the rights rest with the landholders, they would be free to adopt the new land management practices and reap the available benefits, in the process generating more benefits for themselves than the costs they generate for others. On the other hand, if the rights rest with the public, landholders would need to pay for the right to pollute.

The government could potentially choose to allocate the rights to the public throughout the area where public net benefits are negative, but this would not alter the environmental outcomes compared to “no action”, since landholders would be willing to compensate the public in the right-hand triangle, and landholders were never going to adopt the new practices in the bottom-left area (as private net benefits are negative). It would, however, alter the distribution of wealth, since landholders wishing to implement projects in the right-hand triangle would need to pay some of the resulting benefits to the public.

If the environmental manager is using a change in property rights to generate the negative incentives in the lower triangle (labelled “Negative incentives”), it may be practically difficult or expensive to avoid doing the same thing in the “No action” triangle, and perhaps also in the bottom-left “No action” area as well. If this occurs then, as before, it would not alter the environmental outcomes (e.g. polluters would prefer to compensate pollutees in the right “No action” triangle and continue polluting), but it would alter the distribution of benefits between landholders and the public. It would also involve costs from monitoring and enforcement for a much larger area, which would need to be weighed up when considering the overall merits of the approach.

Property rights approaches, as discussed in the last two paragraphs, are relatively flexible, in that they cause external effects of their actions to be felt by landholders, while leaving the ultimate decisions to them. The landholders can therefore weigh up whether the private net benefits outweigh the public net costs when reaching their management decisions. Mechanisms in this category include tradeable pollution permits, pollution taxes, subsidies and conservation auctions.

If a less flexible mechanism, such as command-and-control regulation, is used across the whole area, there would probably be significant net costs in the right “No action” triangle, as landholders would be prevented from doing things that yield relatively large net benefits to them, and relatively small net costs to the public. Thus we would at least partly (and perhaps fully) offset any net benefits generated in the negative incentives area.

Finally I note that the recommendations in Figure 2 (like all of the recommendations in the framework) depend on the landholders having reasonably accurate perceptions about the private net benefits of adoption. If this is not true, there may be roles for extension, positive incentives or negative incentives in other parts of Figure 1.

David Pannell, The University of Western Australia