214 – The perils of asking people

Most of us answer survey questionnaires now and then. If we take a survey seriously, we probably feel that the responses we provide capture how we really feel. Most of us are unaware of how sensitive our answers can be to the way a question is asked.

There are various reasons why it can be a good idea to ask the public about their beliefs, their preferences, their likely behaviour, and so on. Social scientists, including economists, do it regularly. Reasons include to help evaluate policies or projects, or just to understand people. Governments also do it, to help decide which policies to adopt, and how to implement them.

Most people who are not social scientists have no idea how tricky this task of asking people can be. Here is an illustration.

Suppose that the government wishes to foster water conservation by encouraging the adoption of a package of water saving technologies in the home. People who adopt these measures will have lower water bills because of lower water consumption, but the government wants to provide an additional push, and provides a one-off discount on the water bill for households that install them.

Proposal 1: Would you support a proposal that rich households (e.g. with family income over $300,000 per year) should receive a higher discount than poor households (e.g. with family income below $50,000 per year)? Assume that the level of water saving is the same with or without this proposal.

If you are like most people, you would not support that proposal. To most of us, it seems unreasonable to favour rich people. They are already better off, so why should they get an additional benefit?

Now suppose instead that the government takes a different approach to the scheme. In that first scheme, the default position was that the household did not adopt the technologies, and a discount was provided if they did. In the alternative scheme, the view is that everybody should install the technologies, so the default position is that every household does. Everybody receives a one-off reduction in their water bill (the same reduction for everybody, consistent with your preference under the first proposal) to help offset the installation costs. Those households that cannot demonstrate that they have installed the technologies will be hit with a surcharge on their bill.

Proposal 2: Would you support a proposal that poor households should pay just as large a surcharge as rich households? Assume that the level of water saving is the same with or without this proposal.

You probably reject that proposal too. It feels unreasonable, doesn’t it?

If you are one of the majority who answers ‘no’ to both proposals, the bad news is that this is completely illogical. Saying ‘no’ to the first proposal is exactly equivalent to saying ‘yes’ to the second, and answering ‘no’ to the second proposal is equivalent to answering ‘yes’ to the first proporal.

To illustrate: if you say ‘no’ to the first proposal, you are saying that the savings on the water bills of poor people from installing the technology should be just as large as for rich people.

If you say ‘no’ to the second proposal, you are saying that the savings on the water bills of poor people from installing the technology should be less than for rich people. (They would get charged less for not installing, so the cost avoided if they do install would be less.)

(If you are finding it hard to get your head around that, there is a numerical example at the end that should make it clear.)

Why did you answer so illogically? Most people can’t answer this question. They just feel confused.

If you were in the illogical majority and answered ‘no’ to both proposals, go back and look again at both proposals and decide which one do you want to change to ‘yes’.

The amazing thing is that you probably don’t want to change either answer. They both still feel absolutely right, even though you now know that one gives the opposite result to the other.

If we did a public survey on this, we would get completely different results depending on how we described the system – as based on discounts or surcharges.

So which result would be the true representation of how people feel about these issues? My proposed answer to this question is neither/both. The worrying thing for a researcher is that the two contradictory outcomes can both be an accurate reflection of how people feel, depending on the wording and context of the question. It’s as if the real world changes depending on how you ask the question.

This dependence of people’s answers on the wording and context of questions is called a ‘framing effect’. Skilled survey developers who wish to get reasonably accurate results are well aware of the risk of framing effects (although sometimes it is hard to decide which frame is preferred, as in the above example). Less skilled/experienced survey developers – perhaps people who are not social scientists and are just having a go at a survey – often make awful mistakes.

Where does that leave us? Amongst other things, I think it means that we need to be cautious and humble in interpreting the results of surveys and other forms of public consultation. Remember that the way that people express their preferences and beliefs can sometimes be highly sensitive to how the question is asked, and be prepared to take any particular survey result with at least a pinch of salt. I always like to see the exact wording of a survey before I put too much weight on its findings.

It also highlights the importance of not taking the task of developing a survey lightly. If you are responsible for a survey, make sure you use an expert or get expert help and become well-informed about the pitfalls.

Numerical example

Water bills before policy change: Poor people $300; Rich people $500

Scheme 1 (subsidy)

Water bills if we answer ‘no’ to proposal 1, and the subsidy to both groups is $100.

  • If they don’t adopt: Poor $300; Rich $500
  • If they do adopt: Poor $200; Rich $400

Scheme 2 (surcharge)

The discount given to everyone, on the assumption that everyone adopts, is $100. If we answer ‘yes’ to proposal 2, the surcharge to both groups for not adopting is $100.

  • If they don’t adopt: Poor $300; Rich $500
  • If they do adopt: Poor $200; Rich $400

Notice that all the bills are the same as before, whether they adopt or not. So answering ‘no’ to proposal 1 is identical to answering ‘yes’ to proposal 2. If we answer ‘no’ to proposal 2, the bills for people who don’t adopt might be, say, $250 and $500, so rich people get a larger benefit from adopting. That corresponds to answering ‘yes’ to proposal 1.

Further reading

Bütler, M. and Maréchal, M.A. (2007). Framing effects in political decision making: evidence from a natural voting experiment, University of St. Gallen Department of Economics working paper, number 2007-04. IDEAS page for this paper

Kahneman, D. (2011). Thinking, Fast and Slow, Farrar, Straus and Giroux, New York.

Kragt, M. & Bennett, J. W. (2012). Attribute framing in choice experiments: How do attribute level descriptions affect value estimates? Environmental and Resource Economics, 51, 43-59. IDEAS page for this paper

Pannell, P.B.W. and Pannell, D.J. (1999). Introduction to Social Surveying: Pitfalls, Potential Problems and Preferred Practices, SEA Working Paper 99/04.

2 Comments

  • 8 May, 2012 - 12:49 pm | link

    I got an email from Ian Herford:

    G’day Dave

    Interesting example of the perils of framing and interpreting survey questions. I note that you generously assume that those conducting the survey really want to find the truth. This is of course not always so. The well known example from Yes Minister is one of my all-time favourite dialogues:

    Survey one

    Sir Humphrey Appleby: Mr. Woolley, are you worried about the rise in crime among teenagers?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Do you think there is lack of discipline and vigorous training in our Comprehensive Schools?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Do you think young people welcome some structure and leadership in their lives?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Do they respond to a challenge?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Might you be in favour of reintroducing National Service?
    Bernard Woolley: Er, I might be.
    Sir Humphrey Appleby: Yes or no?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Of course, after all you’ve said you can’t say no to that. On the other hand, the survey can reach opposite conclusions.

    Survey two

    Sir Humphrey Appleby: Mr. Woolley, are you worried about the danger of war?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Are you unhappy about the growth of armaments?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Do you think there’s a danger in giving young people guns and teaching them how to kill?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Do you think it’s wrong to force people to take arms against their will?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: Would you oppose the reintroduction of conscription?
    Bernard Woolley: Yes.
    Sir Humphrey Appleby: There you are, Bernard. The perfectly balanced sample.

  • 16 January, 2017 - 2:40 pm | link

    Youtube video for the above excerpt from Yes Minister:
    https://www.youtube.com/watch?v=G0ZZJXw4MTA

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    Please solve this to show you\'re a real person *