Category Archives: Research

279 – Garbage in, garbage out?

As the developer of various decision tools, I’ve lost track of the number of times I’ve heard somebody say, in a grave, authoritative tone, “a model is only as good as the information you feed into it”. Or, more pithily, “garbage in, garbage out”. It’s a truism, of course, but the implications for decision makers may not be quite what you think.

The value of the information generated by a decision tool depends, of course, on the quality of input data used to drive the tool. Usually, the outputs from a decision tool are less valuable when there is poor-quality information about the inputs than when there is good information.

But what should we conclude from that? Does it mean, for example, that if you have poor quality input information you may just as well make decisions in a very simple ad hoc way and not worry about weighing up the decision options in a systematic way? (In other words, is it not worth using a decision tool?) And does it mean that it is more important to put effort into collecting better input data rather than improving the decision process?

No, these things do not follow from having poor input data. Here’s why.

Imagine a manager looking at 100 projects and trying to choose which 10 projects to give money to. Let’s compare a situation where input data quality is excellent with one where it is poor.

decision_aheadFrom simulating hundreds of thousands of decisions like this, I’ve found that systematic decision processes that are consistent with best-practice principles for decision making (see Pannell 2013) do a reasonable job of selecting the best projects even when there are random errors introduced to the input data. On the other hand, simple ad hoc decision processes that ignore the principles often result in very poor decisions, whether the input data is good, bad or indifferent.

Not every decision made using a sound decision process is correct, but overall, on average, they are markedly better than quick-and-dirty decisions. So “garbage in, garbage out” is misleading. If you look across a large number of decisions (which is what you should do), then a better description for a good decision tool could be “garbage in, not-too-bad out”. On the other hand, the most apt description for a poor decision process could be “treasure or garbage in, garbage out”.

An interesting question is, if you are using a good process, why don’t random errors in the input data make a bigger difference to the outcomes of the decisions? Here are some reasons.

Firstly, poorer quality input data only matters if it results in different decisions being made, such as a different set of 10 projects being selected. In practice, over a large number of decisions, the differences caused by input data uncertainty are not as large as you might expect. For example, in the project-selection problem, there are several reasons why data uncertainty may have only a modest impact on which projects are selected:

  • Uncertainty doesn’t mean that the input data for all projects is wildly inaccurate. Some are wildly inaccurate, but some, by chance, are only slightly inaccurate, and some are in between. The good projects with slightly inaccurate data still get selected.
  • Even if the data is moderately or highly inaccurate, it doesn’t necessarily mean that a good project will miss out on funding. Some good projects look worse than they should do as a result of the poor input data, but others are actually favoured by the data inaccuracies, so of course they still get selected. These data errors that reinforce the right decisions are not a problem.
  • Some projects are so outstanding that they still seem worth investing in even when the data used to analyse them is somewhat inaccurate.
  • When ranking projects, there are a number of different variables to consider (e.g. values, behaviour change, risks, etc.). There is likely to be uncertainty about all of these to some extent, but the errors won’t necessarily reinforce each other. In some cases, the estimate of one variable will be too high, while the estimate of another variable will be too low, such that the errors cancel out and the overall assessment of the project is about right.

So input data uncertainty means that some projects that should be selected miss out, but many good projects continue to be selected.

Even where there is a change in project selection, some of the projects that come in are only slightly less beneficial than the ones that go out. Not all, but some.

Putting all that together, inaccuracy in input data only changes the selection of projects for those projects that: happen to have the most highly inaccurate input data; are not favoured by the data inaccuracies; are not amongst the most outstanding projects anyway; and do not have multiple errors that cancel out. Further, the changes in project selection that do occur only matter for the subset of incoming projects that are much worse than the projects they displace. Many of the projects that are mistakenly selected due to poor input data are not all that much worse than the projects they displace. So input data uncertainty is often not such a serious problem for decision making as you might think. As long as the numbers we use are more-or-less reasonable, results from decision making can be pretty good.

To me, the most surprising outcome from my analysis of these issues was the answer to the second question: is it more important to put effort into collecting better input data rather than improving the decision process?

As I noted earlier, the answer seems to be “no”. For the project choice problem I described earlier, the “no” is a very strong one. In fact, I found that if you start with a poor quality decision process, inconsistent with the principles I’ve outlined in Pannell (2013), there is almost no benefit to be gained by improving the quality of input data. I’m sure there are many scientists who would feel extremely uncomfortable with that result, but it does make intuitive sense when you think about it. If a decision process is so poor that its results are only slightly related to the best possible decisions, then of course better information won’t help much.

Further reading

Pannell, D.J. and Gibson, F.L. (2014) Testing metrics to prioritise environmental projects, Australian Agricultural and Resource Economics Society Conference (58th), February 5-7, 2014, Port Macquarie, Australia. Full paper

Pannell, D.J. (2013). Ranking environmental projects, Working Paper 1312, School of Agricultural and Resource Economics, University of Western Australia. Full paper

273 – Behaviour change comes in pairs

Some key factors that drive adoption of new practices come in pairs: one aspect related to the performance of the new practice, and one aspect related to how much people care about that performance. Many models of adoption miss this, including famous ones.

Whatever work or hobbies we do, there are regularly new practices coming along that we are encouraged to adopt: new technologies (e.g. a new iPhone, an auto-steer crop harvester), or different behaviours (e.g. reducing our usage of energy or water, changing the allocation of land to different crops).

The agricultural examples above reflect that some of my research is on adoption of new practices by farmers, but the issue I’m talking about today is relevant in all spheres where people adopt new practices.

It is well recognised that people vary in the personal goals that drive their choices about whether to adopt new practices that are promoted to them. Amongst commercial farmers, for example, there are differences in the emphases they give to profit, risk and environmental outcomes.

Any attempt to understand or model adoption of new practices needs to recognise the potential importance of these different goals. Many studies do include variables representing these three goals, and sometimes others.

However, it is less often recognised that there are two aspects to each of these goals when looking at a new practice:

  1. The extent to which the new practice would deliver the outcome measured by that goal: more profit, less risk, or better environmental outcomes.
  2. How much the decision maker cares about those particular outcomes.

These two aspects are closely linked. They interact to determine how attractive a new practice is, but they are distinctly different. One is not a proxy for the other.

extension 1For example, suppose a farmer is considering two potential new practices for weed control. The farmer judges that new practice A is much riskier (less reliable) than new practice B.

How much will this affect the farmer’s decision making? That depends on the farmer’s attitude to risk. For a farmer who has a strong aversion to risk, practice B will be strongly favoured, at least from the risk perspective. (Other goals will probably also come into play as well.) For a farmer who doesn’t care about risk one way or the other, the difference in riskiness between practices A and B is of no consequence. Some farmers (a minority) have been found to be risk-seeking, so they would prefer practice A.

The same sort of pattern occurs with other goals as well. The attractiveness of a new practice depends on how much difference it makes to profit and on how strongly the farmer is motivated by profit. Or how much it affects the environment and how strongly the farmer cares about the environment.

Amongst the thousands of research studies of farmer adoption of new practices, most represent only one goal-related variable where two are needed. For example, they include a measure of risk aversion, but ignore differences in the level of riskiness of the new practice amongst different adopters. Or they represent differences in the profitability of the new practice, but not differences in how much the adopters care about profit.

It doesn’t help that the issue is not recognised in common conceptual frameworks used by social scientists studying adoption behaviour, such as the Theory of Reasoned Action (Fishbein and Ajzen 1975) and the Theory of Planned Behaviour (Ajzen 1991).

It should be recognised in a sound economics framework (e.g. Abadi Ghadim and Pannell 1999 do so for risk), but it often isn’t included in the actual numerical model that is estimated.

The only framework I’ve seen that really captures this issue properly is our framework for ADOPT – the Adoption and Diffusion Outcome Prediction Tool. Hopefully this insight can diffuse to other researchers over time.

Further reading

Abadi Ghadim, A.K. and Pannell, D.J. (1999). A conceptual framework of adoption of an agricultural innovation, Agricultural Economics 21, 145-154. Journal web page ◊ IDEAS page

Ajzen, I. (1991). The theory of planned behavior, Organizational Behavior and Human Decision Processes 50, 179-211.

Fishbein, M. and Ajzen, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley.

259 – Increasing environmental benefits

It is obvious that the budgets of our public environmental programs are small relative to the cost of fixing all of our environmental problems. If we want to achieve greater environmental benefits from our public investments, what, in broad terms, are the options?

I remember seeing a graph last year – I think it was from the Australian Bureau of Statistics – showing the level of concern felt by the Australian community about environmental issues. It looked to have peaked a few years ago, and was pretty flat, or slightly declining. In that context, the prospects for a big increase in environmental spending over time don’t look good, particularly given the general tightness of government budgets. So I was wondering, if we wanted to double the environmental values protected or enhanced by our public programs, what are the options? I was able to identify several. I’ll list them here, and briefly comment on their potential effectiveness, cost and political feasibility.

  1. Double the budget. Effectiveness: high (in the sense that we could actually double the environmental benefits generated). Cost: high. Politics: very unlikely in the foreseeable future. It wouldn’t be my first priority, anyway. Increasing the budget would be more effective if we first delivered some of the strategies below.
  2. Improve the prioritisation of environmental investments. Improve the usage of evidence, the quality of decision metrics (Pannell 2013), and the quality of evaluation of proposals. Effectiveness: high (because most programs currently have major deficiencies in these areas). Cost: low, especially relative to doubling the budget. Politics: Implies a higher degree of selectivity, which some stakeholders dislike. Probably means funding fewer, larger projects. Achievable for part of the budget but the politics probably require a proportion to be spent along traditional lines (relatively unprioritised).
  3. murray_riverEncourage more voluntary pro-environmental action through education, persuasion, peer pressure and the like. Effectiveness: commonly low, moderate in some cases. Cost: moderate. Politics: favourable.
  4. Increase the share of environmental funds invested in research and development to create pro-environmental technologies (Pannell 2009). Note that this is about creation of new technologies, rather than information. Examples could include more effective baits for feral cats, new types of trees that are commercially viable in areas threatened by dryland salinity, or new renewable energy technologies. Feasibility: case-specific – high in some cases, low in others. Cost: moderate. Politics: requires a degree of patience which can be politically problematic. Also may conflict with community desire to spend resources directly on on-ground works (even if the existing technologies are not suitable). There tends to be a preference for research funding to come from the research budget rather than the environment budget, although this likely means that it is not as well targeted to solve the most important environmental problems.
  5. Improve the design of environmental projects and programs. Improve evidence basis for identifying required actions. Improve selection of delivery mechanisms. Improve the logical consistency of projects. Effectiveness: high (because a lot of existing projects are not well founded on evidence, and/or don’t use appropriate delivery mechanisms, and/or are lacking in internal logical consistency). Cost: low. Politics: Implies changes in the way that projects are developed, with longer lead times, which may not be popular. There may be a perception of high transaction costs from this strategy (although they would be low relative to the benefits) (Pannell et al. 2013).
  6. Increase the emphasis on learning and using better information. Strategies include greater use of detailed feasibility studies, improved outcome-oriented monitoring, and active adaptive management. Effectiveness: moderate to high. Would feed into, and further improve, options 2 and 5. Cost: low. Politics: main barrier is political impatience, and a view that decisions based on judgement are sufficient even in the absence of good information. Often that view is supported/excused by an argument that action cannot and should not wait (which is a reasonable argument in certain cases, but usually is not).
  7. Reform inefficient and environmentally damaging policies and programs. Examples include subsidies for fossil fuels, badly designed policies supporting biofuels in Europe and in the USA, and agricultural subsidies. This strategy is quite unlike the other strategies discussed here, but it has enormous potential to generate environmental benefits in countries that have these types of policies. Successful reform would be not just costless, but cost-saving. Effectiveness: very high in particular cases. Cost: negative. Politics: difficult to very difficult. People with a vested interest in existing policies fight hard to retain them. Environmental agencies don’t tend to fight for this, but there could be great benefits if they did.

In my judgement, for Australia, the top priorities should be strategies 2 and 5 followed by 6. Strategy 4 has good potential in certain cases. If these four strategies were delivered, the case for strategy 1 would be greatly increased (once the politics made that feasible). To succeed, strategies 2, 5 and 6 would need an investment in training and expert support within environmental organisations. Over time, in those environmental organisations that don’t already perform well in relation to strategies 2, 5 and 6 (i.e. most of them), there may be a need for cultural change, which requires leadership and patience.

In Europe and the USA, my first choice would be strategy 7, if it was politically feasible. After that, 2, 5, 6 and 4 again.

Further Reading

Garrick, D., McCann, L., Pannell, D.J. (2013). Transaction costs and environmental policy: Taking stock, looking forward, Ecological Economics 88, 182-184. Journal web site

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Improving environmental decisions: a transaction-costs story, Ecological Economics 88, 244-252. Journal web siteIDEAS page

Pannell, D.J. (2009). Technology change as a policy response to promote changes in land management for environmental benefits, Agricultural Economics 40(1), 95-102. Journal web page ◊ Prepublication version

Pannell, D.J. (2013). Ranking environmental projects, Working Paper 1312, School of Agricultural and Resource Economics, University of Western Australia. IDEAS page ◊ Blog series

234 – The benefits of environmental research

There has been a lot of research on the benefits of research, but little of it has addressed environmental research. In some ways, this is understandable, as it’s difficult. But we need to develop better ways to estimate these benefits as researchers are increasingly asked to justify their funding and quantify their impacts.

I organised a small workshop in Brisbane a few weeks ago on estimating the benefits of environmental research. If we could generate this information, it would be useful in several ways. It could be used to make judgements about whether particular research projects are worth doing, to identify priorities from a set of potential projects, and to make the case for continued funding of environmental research. Also, the process of working out the likely benefits could help us understand the ways that research generates benefits, and that might help us to do a better job of generating benefits.

However, as we quickly agreed at the workshop, this is a very difficult thing to do well. For one thing, there are so many different types of environmental research with different possible uses and impacts, and some of them need different thinking and approaches to analysis.

We decided to focus our attention onto the type of research that is least well served by existing tools and frameworks: research that is intended to influence environmental policy. It turns out that this is the most neglected aspect for a reason – it’s the most difficult one to deal with.

You can see why it’s difficult from the following list of stages that one must go through, starting from research and ending up with real-world benefits.

  • Funding is allocated to research and research is done
  • Something useful is learned – new information is generated (or isn’t)
  • The new information influences policy/management (or doesn’t)
  • Policy change is implemented by policy makers (or isn’t)
  • If the purpose of the policy is to change the behavior of people or businesses, these people respond to the changed policy (or don’t)
  • Changes (relative to no research) result in the environment (or not), including unexpected or unintended consequences

To estimate benefits, we need to estimate what happened (or predict what will happen) at each of these stages. If one link in the chain breaks, benefits are not generated. We also need to estimate (or predict) what would have happened in the absence the research – something you can’t actually observe even if the research has been completed and had its impacts.

Research that aims to influence policy is particularly difficult to assess, because the process of policy change is so complex and influenced by numerous factors. It is very difficult to judge what proportion of any particular change may be attributable to the research rather than other factors. This is recognised in the literature as the attribution problem.

Despite all the difficulties, we found that the existing frameworks for research evaluation provided enough of a platform for us to think productively about what we would do for this type of research. A team of us will be working on this challenge over the next while. We aim to work out what would be needed for a comprehensive rigorous framework, and from that produce a set of principles and perhaps rules of thumb that researchers, research funders and policy makers can use when they need to think about the benefits of policy-oriented environmental research.

Further reading

van der Most, F. (2010). Use and non-use of research evaluation: A literature review, Paper no. 2010/16, Circle, Lund University, Sweden. Here

233 – Journal refereeing

Peer review of research is a key mechanism for quality control used in science. Unfortunately, some reviewers (or referees) perform their task in a hard and heartless way.

Back in 2002 I published a poem about this in a refereed journal article. I’m pretty pleased with this – you don’t see many poems in refereed journals. This week, somebody told me that my poem had been included (with praise such as, “a beautiful piece of work”) on a web page of econometric poetry. I then did a search and, apart from finding the original paper, I found it reproduced on three other pages (here, here, here), and referred to on several more. Isn’t the web marvelous?

In case you haven’t seen it, here it is.

I’m The Referee
David J. Pannell

You’ve posted in your paper
To a journal of repute
And you’re hoping that the referees
Won’t send you down the chute

You’d better not build up a sense of
False security
I’ve just received your manuscript and
I’m the referee

This power’s a revelation
I’m so glad it’s come to me
I can be a total bastard with
Complete impunity

I used to be a psychopath
But never more will be
I can deal with my frustrations now that
I’m a referee

 

The poem is therapeutic, as was the paper it was published in (Pannell, 2002), so if you’ve suffered at the hands of referees, you might want to read that too.

Further reading

Pannell, D.J. (2002). Prose, psychopaths and persistence: personal perspectives on publishing. Canadian Journal of Agricultural Economics 50(2), 101–116. Here ♦ IDEAS page for this paper