Yearly Archives: 2015

282 – MOOC reflections

You’ve probably heard of MOOCs (Massive Online Open Courses). They emerged in 2012 and have grown rapidly in number and popularity. Most MOOCs are free to do, are fairly short, and are provided by universities for public relations purposes. Around 2500 MOOCs have been offered on a huge range of topics. For example, I’ve participated in a MOOC on water management in cities and one on music production and recording.

In late 2013, the University of Western Australia was approached by Yara Pilbarra (a local subsidiary of an international fertilizer company), who were interested in developing a funding partnership. Various possible initiatives were discussed over some months, but eventually it was agreed that I would lead the development and delivery of a MOOC titled “Agriculture, Economics and Nature”. An agreement was signed in August 2014.

Since that MOOC was run in February-March this year, I’ve had a number of people asking about it. How much work was it? What was involved? Was it successful? Was it worth it? Here are some thoughts on these questions.

First some general background about the course. Following discussions with an expert and looking at other successful MOOCs, I decided to make it a six-week course, based around a set of brief video lectures (5-6 minutes long, on average).

Each week, the course included:

  • Between 7 and 10 brief video lectures
  • An interview with an expert who reinforced the material for that week
  • Two or three brief recommended readings, many of which were Pannell Discussions
  • Some optional readings, a few of which were a bit longer or more technical
  • In some weeks, one or two optional videos not created by me
  • A multi-choice quiz (10 questions), which was to reinforce learning, not for assessment

The assessment consisted of a final exam of 60 multi-choice questions.

It was pitched as an introductory unit covering a broad range of relevant and interesting issues. It was something of a sampler, not going into depth on any issue, but providing sufficient information to provide an initial understanding and spark interest. It required no prior background in agriculture or economics. I had in mind that it should be understandable to an intelligent year 12 high school student. Although I worried at times that I had made it too simple and superficial, the responses of participants showed clearly that I had not. Satisfied participants included at least one university professor and a number of post-graduate students.

Thanks to the sponsorship, I was able to employ an assistant to work on collection of materials, preparation of the lectures and creation of slides. I also put a lot of time into this, of course, but the assistance was great.

The money also paid for a much higher quality of video production than would have been possible otherwise. I think this made a real difference to the student experience. About a third of the videos were done in the studio, and the rest were outdoors at various locations, including on farms. Here are a couple of examples.

We put quite a bit of effort into promoting the course through various channels, and ended up with 3200 people enrolling, from the following countries (from most to least number of participants):

Australia, Vietnam, United States, Kenya, Israel, Canada, India, Nigeria, Tunisia, United Kingdom, Brazil, Germany, China, Sri Lanka, Rwanda, St. Lucia, Philippines, New Zealand, Sudan, Jordan, Algeria, Benin, Greece, Ethiopia, Kazakhstan, Ghana, Venezuela, Colombia, Netherlands, South Africa, Georgia, Mexico, Bolivia, Bangladesh, Spain, Armenia, Costa Rica, Malaysia, Brunei, Slovenia, Thailand, Hungary, Sweden, Japan, Singapore, Zambia, Norway, Italy, Guatemala, Oman, Ireland, Tanzania, Nepal, Belgium, Zimbabwe, France, Kosovo, Uganda, Iran, Cameroon, Pakistan, Peru, Isreal, Egypt, Switzerland, Turkey.

One general characteristic of MOOCs is that they have a high drop-out rate. Usually, the completion rate is 10% or less, so I was delighted that 23% of our students completed the final exam. And of course lots more students benefited by completing some or most of the course without doing the final exam.

The amount of work required to create and deliver the course was significant. Here are my estimates of days required over the 18-month duration of the project. Most of the work happened during the 3-4 months of creating and videoing the course and then 6 weeks of running it.

TaskDavid PannellVarious Others
Designing the course and preparing content (slides, graphs, dot points, etc.).10 days50 days
Video production8 days15 days
Selecting readings and supplementary videos, writing material to fill gaps2 days
Preparing quiz and exam questions1.5 days
Setting up the Course2Go web site and uploading all of the materials.2 days
Checking all the materials and links on the Course2Go web site. 1.5 days
Monitoring discussion forums and responding.3 days2 days
Responding to messages and emails.1 day
Project management2 days5 days
Total29 days74 days

Despite the high level of effort required, I consider it to have been worthwhile. It has certainly succeeded in raising the profile of the School of Agricultural Economics and UWA, and many people reported being very interested in the content. A highlight was being recognised on the street in Perth by someone who had done the course! We have seen an increase in inquiries about our normal courses, and many respondents to the course survey expressed interest in receiving information about related courses that we offer.

As a spin-off benefit, I have used a number of the videos in another unit I teach. The experience has also influenced my thinking about how to make regular teaching more engaging and interesting for students.

Apart from all that, I actually enjoyed it. It was a novel experience that allowed me to be creative in different ways, and got me interacting with interesting people from all over the world. It was great to get people’s feedback, which was overwhelmingly positive. For example:

  • The multi-media teaching methodology was excellent. I really appreciated it.
  • Easily accessible information, presented in a format that was informative and also able to progress at my own pace.
  • I learned that economics can be understandable.
  • This was my first MOOC learning experience although I have been involved with on-line tutoring previously. I really enjoyed the pedagogy: being presented with small chunks of information as an overview; clear slides; summaries and then the opportunity to delve deeper with videos and suggested readings. It caters to a wide range of learners’ interests, abilities and time commitments. Basic economics were well explained/revised for me
  • All the lectures were very informative. Being a student of agriculture in Bachelors and Natural Resource Economics in masters this course was very much useful for me from agricultural and environmental point of view. Looking forward to learn more.
  • The videos were well prepared and presented. This is a very good strategy of teaching which included videos, slides, interviews and reading material. The resources provided were excellent. I have learnt a lot as a teacher from Professor David Pannell.
  • The course structure builds up one’s knowledge from the very basic and progresses to more complex things. I very much liked the real life application with case studies highlighted like the Australian Wool crisis, the Gippsland lakes and also the interviews with real stakeholders. It makes one realize that it’s not just theoretical, this is something applicable to everyday life.
  • The overall course layout, starting from basic and building up to useful examples and integrated knowledge around many different concepts. Lastly the interviews really did it for me. Real examples of people practising the theory.
  • The course is well organised and I am sad because it is already finished. I would like to thank your great team.
  • Very interesting course in that it brings to your mind things you thought were difficult seem easy
  • I appreciated the extent to which the professor was prepared to involve himself in the online discussions. I have undertaken many online courses, and professors are often absent from online discussions.
  • The format and duration, not too long and not too short.

We will offer the course again soon, no later than February 2016. To receive details of how to register, email

I presented a webinar on October 13 2015, covering a similar set of issues as this blog post. You can view a video recording of the webinar here:

Further reading

There is more information about the course available here:


281 – Ranking Environmental Projects Revisited

In 2013 I put out a set of 20 blog posts giving practical advice about ranking environmental projects. I’ve upgraded the consolidated report about this to make it even easier to use and more broadly applicable.

My working paper on ranking environmental projects has been quite popular, with about 450 downloads, and I’ve had good feedback from various people who actually used it. I’ve put out a new version of the working paper with lots of small improvements, but with three main ones.

  1. I’ve included two additional ways to estimate the potential benefits of a project. The original paper was based on the approach we use in INFFER, which is designed for projects that address discrete, identifiable environmental assets. There are environmental projects that aren’t like that, and the new report makes it easier to evaluate and rank those.
  2. I’ve provided templates that can be used to ask the specific questions about each project to collect the required information for evaluation and ranking. These are somewhat based on INFFER, but they are Word documents, so that users can easily adapt them to suit their own purposes.
  3. I’ve provided spreadsheets which can be used to collate the information collected in the Word templates. They include the formulas needed to calculate the Benefit: Cost Ratio for each project. There are various versions of the spreadsheets, matching the different approaches outlined in the report. The versions vary in complexity and detail, depending on how much depth the environmental organisation wants to go into for each project.

The new version of the working paper is available at AgEcon Search, and the templates and spreadsheets can be downloaded from the data archive page on my web site.

I’ve also created a version of the report that is specifically targeted to urban water managers and utilities. There are many organisations linked to the CRC for Water Sensitive Cities who I think could find this version useful.

Further Reading

Pannell, D.J. (2015). Ranking environmental projects, Working Paper 1506, School of Agricultural and Resource Economics, University of Western Australia. Full paper ♦ Water Sensitive Cities version

Pannell, D.J., Roberts, A.M., Park, G., Alexander, J., Curatolo, A. and Marsh, S. (2012). Integrated assessment of public investment in land-use change to protect environmental assets in Australia, Land Use Policy 29(2): 377-387. Journal web site ♦ IDEAS page for this paper

280 – Lomborg at UWA

The news of Bjorn Lomborg establishing the “Australian Consensus Centre” at the University of Western Australia has generated plenty of media attention and much discussion within the University.

Some people within UWA are concerned about the University becoming associated with such a controversial and divisive figure. They are worried about the University’s reputation, and about the perception that his work is scientifically flawed.

There has also been commentary on the fact that the Australian Government could find $4 million for this initiative at a time when government funding in general (and university funding in particular) is under such great pressure.

I had no idea that the UWA arrangement was in prospect until Lomborg dropped in to meet me briefly the day before it was announced a couple of weeks ago. I had not had any contact with him in the past. I found him to be very personable and he asked sensible and genuine questions about environmental issues in Australia.

I have been aware of Lomborg’s work since his 2001 book The Skeptical Environmentalist (TSE), with its message that many environmental problems are not as bad as we’d been led to believe, and some are getting better. I’ve also been interested in his writings on climate policy, and his more-recent initiative, the Copenhagen Consensus, which sets out to prioritise a set of major international policies. The latter will be adapted for a new set of policies in the UWA initiative.

tseAll three of these areas of work have generated controversy and criticism. I’ve read many of the critics, particularly in the early days of TSE when the criticism was raging.

I split the criticisms into two types: identification of errors of fact and criticisms of what he does with the facts (interpretations and judgements). There were some errors of fact in TSE, but not as many as claimed. In my judgement, many of the claimed errors were misinterpretations, misunderstandings or misrepresentations by the critics, quoting him out of context, highlighting trivial issues, and so on. People who didn’t like his conclusions went out of their way to find the smallest hint of an error and blow it up. There is a web site called Lomborg Errors, which includes numerous so-called errors from TSE, but when I read it I was singularly unconvinced by many of them. In fact I found myself laughing out loud at some of them. Given the huge scope of the book, the number of significant, genuine errors is remarkable small, really, and they don’t change the general message of the book. But the myth of there being numerous serious errors got well established, and is accepted as received wisdom by many.

I didn’t agree with everything in TSE. Some parts were less convincing, and it seemed too optimistic to me in some respects. These were generally not errors of fact, but differences in judgement about what they implied or what should be done about them.

Lomborg has faced plenty of disagreement about his policy recommendations, particularly in relation to climate change. He argues that the political barriers to pricing carbon at a price level that will achieve the desired outcomes are so great that we may as well not bother with it. Instead he advocates a large public investment in development of new technologies, such as for renewable energy. This position is obviously at odds with most people who are concerned about climate change, but my own view is that his pessimism about the politics is justified (reinforced by the messages coming out of India recently) and that the technology route is likely to be the only approach with any real chance of averting serious climate change. I’ve written about this here.  Interestingly, his position is not that of a climate sceptic/denier, although he is sometimes characterised as being one.

Looking around the web, I see some scientists arguing that climate change will be greater and more costly than Lomborg has concluded in his climate book, often coupled with attribution of dubious motivations and associations. Perhaps he has made errors here and underplayed some potential outcomes – I haven’t taken the time to evaluate the claims. Nevertheless, even if he has, it doesn’t affect the logic behind his recommended policy approach.

The Copenhagen Consensus work, a version of which he will bring to UWA, is somewhat different in nature. His contribution is to set up and manage the process, bring people together and publicise the results. The judgements made in the process are not his judgements, but those of panels of people (usually senior economists) responding to evidence and cases put by commissioned experts. The focus is on identifying priorities for policy action. From a set of defined policies, which are the ones that are likely to have the greatest benefits for mankind? The explicit focus on prioritisation is critical, but is often missed by people advocating for a particular policy.

The controversy here arises because carbon-pricing policies consistently come out as being much lower in priority than other things like improving childhood nutrition in developing countries and fighting infectious diseases. In my view, this result isn’t a surprise, considering the likely benefits, feasibility, time lags and costs of the options. But it adds to the impression that Lomborg is a climate “contrarian”, even though the results are not actually generated by him.

Some have argued that the concept of prioritising these policies is wrong – we should just implement them all. I think that’s very naïve. It’s not how the world works. None of the policies being evaluated is currently in place. It’s a huge, difficult, risky task to try to get a major new policy adopted, especially when international agreements are needed. Governments have to carefully prioritise how to spend their financial resources and their political capital.

It’s very interesting that Vice Chancellor Paul Johnson has signed up to the University hosting this new centre. He must have anticipated that there would be controversy. I think it’s positive that the University hasn’t been scared off. A university is a good place to do work that challenges people to think differently.

Overall, if it can sufficiently avoid the taint of politics (which might be tricky), I think the initiative could make a worthwhile and interesting contribution to the policy debate in Australia. But also there will no doubt be aspersions cast against Lomborg and UWA.

279 – Garbage in, garbage out?

As the developer of various decision tools, I’ve lost track of the number of times I’ve heard somebody say, in a grave, authoritative tone, “a model is only as good as the information you feed into it”. Or, more pithily, “garbage in, garbage out”. It’s a truism, of course, but the implications for decision makers may not be quite what you think.

The value of the information generated by a decision tool depends, of course, on the quality of input data used to drive the tool. Usually, the outputs from a decision tool are less valuable when there is poor-quality information about the inputs than when there is good information.

But what should we conclude from that? Does it mean, for example, that if you have poor quality input information you may just as well make decisions in a very simple ad hoc way and not worry about weighing up the decision options in a systematic way? (In other words, is it not worth using a decision tool?) And does it mean that it is more important to put effort into collecting better input data rather than improving the decision process?

No, these things do not follow from having poor input data. Here’s why.

Imagine a manager looking at 100 projects and trying to choose which 10 projects to give money to. Let’s compare a situation where input data quality is excellent with one where it is poor.

decision_aheadFrom simulating hundreds of thousands of decisions like this, I’ve found that systematic decision processes that are consistent with best-practice principles for decision making (see Pannell 2013) do a reasonable job of selecting the best projects even when there are random errors introduced to the input data. On the other hand, simple ad hoc decision processes that ignore the principles often result in very poor decisions, whether the input data is good, bad or indifferent.

Not every decision made using a sound decision process is correct, but overall, on average, they are markedly better than quick-and-dirty decisions. So “garbage in, garbage out” is misleading. If you look across a large number of decisions (which is what you should do), then a better description for a good decision tool could be “garbage in, not-too-bad out”. On the other hand, the most apt description for a poor decision process could be “treasure or garbage in, garbage out”.

An interesting question is, if you are using a good process, why don’t random errors in the input data make a bigger difference to the outcomes of the decisions? Here are some reasons.

Firstly, poorer quality input data only matters if it results in different decisions being made, such as a different set of 10 projects being selected. In practice, over a large number of decisions, the differences caused by input data uncertainty are not as large as you might expect. For example, in the project-selection problem, there are several reasons why data uncertainty may have only a modest impact on which projects are selected:

  • Uncertainty doesn’t mean that the input data for all projects is wildly inaccurate. Some are wildly inaccurate, but some, by chance, are only slightly inaccurate, and some are in between. The good projects with slightly inaccurate data still get selected.
  • Even if the data is moderately or highly inaccurate, it doesn’t necessarily mean that a good project will miss out on funding. Some good projects look worse than they should do as a result of the poor input data, but others are actually favoured by the data inaccuracies, so of course they still get selected. These data errors that reinforce the right decisions are not a problem.
  • Some projects are so outstanding that they still seem worth investing in even when the data used to analyse them is somewhat inaccurate.
  • When ranking projects, there are a number of different variables to consider (e.g. values, behaviour change, risks, etc.). There is likely to be uncertainty about all of these to some extent, but the errors won’t necessarily reinforce each other. In some cases, the estimate of one variable will be too high, while the estimate of another variable will be too low, such that the errors cancel out and the overall assessment of the project is about right.

So input data uncertainty means that some projects that should be selected miss out, but many good projects continue to be selected.

Even where there is a change in project selection, some of the projects that come in are only slightly less beneficial than the ones that go out. Not all, but some.

Putting all that together, inaccuracy in input data only changes the selection of projects for those projects that: happen to have the most highly inaccurate input data; are not favoured by the data inaccuracies; are not amongst the most outstanding projects anyway; and do not have multiple errors that cancel out. Further, the changes in project selection that do occur only matter for the subset of incoming projects that are much worse than the projects they displace. Many of the projects that are mistakenly selected due to poor input data are not all that much worse than the projects they displace. So input data uncertainty is often not such a serious problem for decision making as you might think. As long as the numbers we use are more-or-less reasonable, results from decision making can be pretty good.

To me, the most surprising outcome from my analysis of these issues was the answer to the second question: is it more important to put effort into collecting better input data rather than improving the decision process?

As I noted earlier, the answer seems to be “no”. For the project choice problem I described earlier, the “no” is a very strong one. In fact, I found that if you start with a poor quality decision process, inconsistent with the principles I’ve outlined in Pannell (2013), there is almost no benefit to be gained by improving the quality of input data. I’m sure there are many scientists who would feel extremely uncomfortable with that result, but it does make intuitive sense when you think about it. If a decision process is so poor that its results are only slightly related to the best possible decisions, then of course better information won’t help much.

Further reading

Pannell, D.J. and Gibson, F.L. (2014) Testing metrics to prioritise environmental projects, Australian Agricultural and Resource Economics Society Conference (58th), February 5-7, 2014, Port Macquarie, Australia. Full paper

Pannell, D.J. (2013). Ranking environmental projects, Working Paper 1312, School of Agricultural and Resource Economics, University of Western Australia. Full paper

278 – Global wealth inequality

The charity Oxfam recently released a remarkable report on international wealth inequality. Based on data and analysis published by the Swiss financial company Suisse Credit, they highlighted that the aggregate wealth of the world’s richest one percent of people is about the same as the aggregates wealth of the other 99 percent.

This made my head spin, so I wanted to see the graph of wealth distribution. Using the Oxfam/Suiss Credit data, I put together an approximation of the Lorenz Curve for the whole world (Figure 1). To create a Lorenz curve, you rank all the people, from poorest to richest, and plot the proportion of the world’s wealth that they own. The graph shows the proportion of the world’s wealth that is owned by the poorest X percent.


Figure 1. The percentage of the world’s wealth that is owned by the poorest X percent of the population.


The figure reinforces the remarkable extent of inequality indicated in the headline 1%:99% fact.

For example, it shows that the least-wealthy 70% of people own just a few percent of the world’s wealth between them.

90% of people have a bit more than 10% of the wealth.

The wealth of the bottom 30% is roughly zero. If you look closely, you can see that the line disappears below the axis for the bottom group of people, indicating that they have slightly negative wealth.

At the other extreme, the wealth of the very richest people is astounding. You can’t make this out on the graph, but the richest 80 people in the world – with individual wealth ranging from $13 billion to $76 billion in 2014 – have as much wealth between them as the bottom 50% of people on the planet. That’s 80 people versus 3,500,000,000 people.

However, you might be surprised to learn that the story of the richest 1 percent is not all about billionaires, or even millionaires. To make it into the richest 1 percent, you need wealth of about $800,000. There are 1.8 million such people in Australia. Those of us who live in Australia (or in any developed country) would come across top 1 percenters on a regular basis – they are all around us. They are mostly not people living a jet-set lifestyle. Within a developed-country context, most of them would not be considered especially rich.

That is even more true of the top 10 percent. The wealth you need to make it into that group is only $77,000. As one of my colleagues commented, this reveals that the problem is not “those rich bastards”. It’s us!

slumsThis is not to say that the poor are not improving their lot. In many developing countries, the average wealth of poor people, and especially middle-ranked people, has improved over time (see here). It’s just that the wealth of people who are already wealthy is growing more rapidly, not just absolutely but relatively.

Another surprising result is that there are quite a few people from developed countries at the bottom end of the distribution. These are mostly people who have assets, and actually have a pretty good standard of living, but they also have large debts that leave them with negative net wealth. The collapse of house prices in the US associated with the Global Financial Crisis created many such people. Remarkably, about 7% of Americans are in the bottom 10% for net wealth. Only India has more people in this poorest group! Of course, this reveals that net wealth is not the whole story. An American from the bottom 10% is likely to have a much higher standard of living and much greater opportunities for improvement than an Indian from the bottom 10%.

The difficult thing, of course, is the question of what should be done about all this inequality. Oxfam has some proposals, but others have argued that inequality per se is not a problem, as long as the lot of the poor is improving. To me it seems that extreme inequality is a concern in its own right, particularly within a country, but that it would be hard to support measures to dampen inequality if doing so would make poor people worse off. This is a  can of worms, of course.

Further reading

Bellù, L.G. and Liberati, P. (2005). Social Welfare Analysis of Income Distributions: Ranking Income Distributions with Lorenz Curves, IDEAS page.

Credit Suisse (2014). Global Wealth Data Book, online here.

Oxfam (2015). Wealth: Having It All and Wanting More, online here.

News reports: here, here, herehere