Monthly Archives: March 2013

234 – The benefits of environmental research

There has been a lot of research on the benefits of research, but little of it has addressed environmental research. In some ways, this is understandable, as it’s difficult. But we need to develop better ways to estimate these benefits as researchers are increasingly asked to justify their funding and quantify their impacts.

I organised a small workshop in Brisbane a few weeks ago on estimating the benefits of environmental research. If we could generate this information, it would be useful in several ways. It could be used to make judgements about whether particular research projects are worth doing, to identify priorities from a set of potential projects, and to make the case for continued funding of environmental research. Also, the process of working out the likely benefits could help us understand the ways that research generates benefits, and that might help us to do a better job of generating benefits.

However, as we quickly agreed at the workshop, this is a very difficult thing to do well. For one thing, there are so many different types of environmental research with different possible uses and impacts, and some of them need different thinking and approaches to analysis.

We decided to focus our attention onto the type of research that is least well served by existing tools and frameworks: research that is intended to influence environmental policy. It turns out that this is the most neglected aspect for a reason – it’s the most difficult one to deal with.

You can see why it’s difficult from the following list of stages that one must go through, starting from research and ending up with real-world benefits.

  • Funding is allocated to research and research is done
  • Something useful is learned – new information is generated (or isn’t)
  • The new information influences policy/management (or doesn’t)
  • Policy change is implemented by policy makers (or isn’t)
  • If the purpose of the policy is to change the behavior of people or businesses, these people respond to the changed policy (or don’t)
  • Changes (relative to no research) result in the environment (or not), including unexpected or unintended consequences

To estimate benefits, we need to estimate what happened (or predict what will happen) at each of these stages. If one link in the chain breaks, benefits are not generated. We also need to estimate (or predict) what would have happened in the absence the research – something you can’t actually observe even if the research has been completed and had its impacts.

Research that aims to influence policy is particularly difficult to assess, because the process of policy change is so complex and influenced by numerous factors. It is very difficult to judge what proportion of any particular change may be attributable to the research rather than other factors. This is recognised in the literature as the attribution problem.

Despite all the difficulties, we found that the existing frameworks for research evaluation provided enough of a platform for us to think productively about what we would do for this type of research. A team of us will be working on this challenge over the next while. We aim to work out what would be needed for a comprehensive rigorous framework, and from that produce a set of principles and perhaps rules of thumb that researchers, research funders and policy makers can use when they need to think about the benefits of policy-oriented environmental research.

Further reading

van der Most, F. (2010). Use and non-use of research evaluation: A literature review, Paper no. 2010/16, Circle, Lund University, Sweden. Here

233 – Journal refereeing

Peer review of research is a key mechanism for quality control used in science. Unfortunately, some reviewers (or referees) perform their task in a hard and heartless way.

Back in 2002 I published a poem about this in a refereed journal article. I’m pretty pleased with this – you don’t see many poems in refereed journals. This week, somebody told me that my poem had been included (with praise such as, “a beautiful piece of work”) on a web page of econometric poetry. I then did a search and, apart from finding the original paper, I found it reproduced on three other pages (here, here, here), and referred to on several more. Isn’t the web marvelous?

In case you haven’t seen it, here it is.

I’m The Referee
David J. Pannell

You’ve posted in your paper
To a journal of repute
And you’re hoping that the referees
Won’t send you down the chute

You’d better not build up a sense of
False security
I’ve just received your manuscript and
I’m the referee

This power’s a revelation
I’m so glad it’s come to me
I can be a total bastard with
Complete impunity

I used to be a psychopath
But never more will be
I can deal with my frustrations now that
I’m a referee


The poem is therapeutic, as was the paper it was published in (Pannell, 2002), so if you’ve suffered at the hands of referees, you might want to read that too.

Further reading

Pannell, D.J. (2002). Prose, psychopaths and persistence: personal perspectives on publishing. Canadian Journal of Agricultural Economics 50(2), 101–116. Here ♦ IDEAS page for this paper