325 – Ranking projects based on cost-effectiveness

Where organisations are unable or unwilling to quantify project benefits in monetary or monetary-equivalent terms, a common approach is to rank potential projects on the basis of cost-effectiveness. Just like ranking projects based on Benefit: Cost Ratio (BCR), this approach works in some cases but not others.

To rank projects based on cost-effectiveness, you choose the metric you will use to measure project benefits, estimate that metric for each project, estimate the cost of each project, and divide the benefit metric by the cost. You end up with a cost-effectiveness number for each potential project, and you use these numbers to rank the projects.

An advantage of this approach is that it sidesteps the challenges of having to measure all the benefits in monetary or monetary-equivalent terms, which is what you have to do calculate a BCR. A disadvantage is that it only works to compare projects that generate similar types of benefits, which can all be measured with the same metric.

Assuming that we are satisfied with your benefits metric and that the projects to be ranked are similar enough, the question is, in what circumstances is it appropriate to rank projects based on cost-effectiveness? (Assuming that the objective is to maximise the overall benefits across all the projects that get funded.) It is logical to ask this given that cost-effectiveness is closely related to the BCR (it has the same structure – it’s just that benefits are measured differently), and we’ve seen in PD322, PD323 and PD324 that ranking projects by BCR works in some situations but not others.

It turns out that the circumstances where it is logical to use cost-effectiveness to rank projects are equivalent to the circumstances where it is logical to rank projects using BCR.

(i) If you are ranking separate, unrelated projects, doing so on the basis of cost-effectiveness is appropriate. Ranking projects by cost-effectiveness implies that there is a limited budget available and you are aiming to allocate it to the best projects.

(ii) If you are ranking mutually exclusive projects (e.g. different versions of the same project), ranking on the basis of cost-effectiveness can be highly misleading. If there are increasing marginal costs and/or decreasing marginal benefits (which are normal), ranking by cost-effectiveness will bias you towards smaller project versions. In PD323, I said to rank such projects by NPV and choose the highest NPV you can afford with the available budget. If we are not monetising the benefits, there is no equivalent to the NPV — you cannot subtract the costs from a non-monetary version of the benefits. This means that, strictly speaking, you cannot rank projects in this situation (mutually exclusive projects) without monetising the benefits. If you absolutely will not or cannot monetise the benefits, what I suggest you do instead is identify the set of project versions that can be afforded with the available budget, and choose the project version from that set that has the highest value for the benefit metric. You don’t divide by the costs, but you do use the costs to determine which project versions you can afford. This is a fudge that only makes sense if you adopt the unrealistic assumption that any unspent money will not be available to spend on anything else, but it seems to me to be the best way to go, if monetising the benefits is not an option.

(iii) If you are ranking separate, unrelated projects, and there are multiple versions available for at least one of those projects, then cost-effectiveness does not work and the rule about choosing the highest-value benefit metric does not work either. Instead, you should build an integer programming model to simultaneously weigh up both problems: which project(s) and which project version(s). There is a brief video showing you how to do this in Excel in PD324. In the video, the benefits are measured in monetary terms, but the approach will work if you use non-monetary measures of the benefits.

There are a number of tools available for ranking projects based on cost-effectiveness (e.g. Joseph et al. 2009) but it is important to be clear that the approach only works in certain cases.

Even if you are using cost-effectiveness in the right circumstances (case (i) above), it has a couple of limitations relative to using BCR. One is that you cannot use it to rank projects with distinctly different types of benefits that cannot all be measured with the same metric. Another limitation is that cost-effectiveness provides no evidence about whether any of the projects would generate sufficient benefits to outweigh its costs.

Further reading

Joseph, L.N., Maloney, R.F. and Possingham, H.P. (2009). Optimal allocation of resources among threatened species: a project prioritization protocol. Conservation Biology, 23, 328-338.  Journal web site

Pannell, D.J. (2015). Ranking environmental projects revisited. Pannell Discussions 281. Here * IDEAS page

324 – NPV versus BCR part 3

In PD322 and PD323 we have been exploring whether to use Net Present Value (NPV) or Benefit: Cost Ratio (BCR) in Benefit: Cost Analysis (BCA) when assessing and comparing projects. I’ve presented some simple rules to follow when the projects are separate and unrelated, or when the projects are mutually exclusive (i.e., when you can only choose to do one of them). But what if the decision maker is faced with choosing from multiple separate projects, and there are multiple versions of at least one of the projects?

Before getting into the details, I want to clarify that, in all the examples I’m presenting in this series, I’m assuming that the objective is to maximise the total NPV across all funded projects. I should have emphasised that earlier. A key message from the three related blog posts is that to get the highest total NPV, you should not necessarily choose the projects with the highest individual NPVs. Sometimes that’s the case but in other cases it’s not.

Now, suppose that a decision maker is faced with selecting from three different versions of a project to protect Lake Antelope (call them projects A1, A2 and A3) and four versions of a project to protect Lake Giraffe (projects G1, G2, G3 and G4). In this situation, there is no simple rule to follow, like just choose the projects with the highest BCRs or the highest NPVs. Instead, this situation requires the use of a constrained optimisation algorithm. Assuming that you are choosing whole projects (i.e. you can’t choose 0.7 of a project), the required algorithm is called integer programming.

Fortunately, integer programming is available in the Excel spreadsheet software, and it is not difficult to use it to select the optimal portfolio of projects in this complex situation. I’ll present the numbers for a relatively simple example of this complex decision problem, and then I’ll show you how to solve it in a brief YouTube video.

First the example. The present values of benefits (B) and costs (C) for the project versions for Lake Antelope and Lake Giraffe are as follows: project A1: B=$180, C=$40; project A2: B=$360, C=$100; project A3: B=$400, C=$200; project G1: B=$200, C=$10; project G2: B=$400, C=$60; project G3: B=$600, C=$160; project G4: B=$800, C=$310. The available budget is $300. You can choose at most one of the Lake Antelope project versions and at most one of the Lake Giraffe project versions. Which project versions (if any) should you choose to maximise the overall net benefits?

Watch the video to see how to use integer programming to solve this in Excel. Within the capacity of the software, this approach will work for any number of projects and any number of project versions.

Note that the information you get from a model like this is not a simple ranking of the projects. Instead, it tells you how the optimal combination of projects and project versions changes depending on the available budget. To get this information, you change the program budget in the model and re-solve it. For this example, the results look like this.

Range of budget levelsOptimal version of A projectOptimal version of G project
$0 to $9nilnil
$10 to $49nilG1
$50 to $59A1G1
$60 to $99nilG2
$100 to $159A1G2
$160 to $259A2G2
$260 to $409A2G3
$410 and higherA2G4

 

The two project versions in each row of the table come as a package. There is no point in ranking them. If the budget was to shrink, you would not just drop one of these, you would change which project version you selected, as shown in the table.

NPV/BCR Rule 4: If selecting and ranking multiple projects, and at least one of the projects is available in multiple versions, don’t use NPV or BCR. Instead, use integer programming to optimise the selection of projects and project versions simultaneously.

Having to create a constrained optimisation model like this might seem like more bother than is worthwhile, but it isn’t difficult, it doesn’t take long, and it may result in substantially higher benefits being generated by the program, compared with the use of a simpler rule-of-thumb.

If you’ve read all three of these Pannell Discussions on NPV and BCR, your head may be spinning at the complexity of all this. Apologies for that, but the complexity is real and needs to be understood if analysts applying BCA are to be sure of giving sound advice to decision makers.

Appendix

Another realistic complexity that might be relevant in some cases is that there might be two separate constraints on the funding of projects. For example, there could be a limited budget available for initial project implementation and a separate limited budget available for ongoing maintenance. As with the example above, neither BCR nor NPV is sure to rank the projects correctly in this situation, and you need to use a method like integer programming to be sure of getting it right. You could build an Excel model like the one in the video, and include separate constraints for implementation costs and maintenance costs, rather than a single constraint for costs overall, which is what I did in the video.

On the other hand, in the numerical examples I’ve looked at, it is usually not too terrible to assume that implementation costs and maintenance costs are drawn from one limited budget. In other words, using BCR to rank separate, unrelated projects can still be OK even if there are two constraints on funding. But it is an approximation and it might not give you the best possible solution.