325. Ranking projects based on cost-effectiveness
Where organisations are unable or unwilling to quantify project benefits in monetary or monetary-equivalent terms, a common approach is to rank potential projects on the basis of cost-effectiveness. Just like ranking projects based on Benefit: Cost Ratio (BCR), this approach works in some cases but not others.
To rank projects based on cost-effectiveness, you choose the metric you will use to measure project benefits, estimate that metric for each project, estimate the cost of each project, and divide the benefit metric by the cost. You end up with a cost-effectiveness number for each potential project, and you use these numbers to rank the projects.
An advantage of this approach is that it sidesteps the challenges of having to measure all the benefits in monetary or monetary-equivalent terms, which is what you have to do calculate a BCR. A disadvantage is that it only works to compare projects that generate similar types of benefits, which can all be measured with the same metric.
Assuming that we are satisfied with your benefits metric and that the projects to be ranked are similar enough, the question is, in what circumstances is it appropriate to rank projects based on cost-effectiveness? (Assuming that the objective is to maximise the overall benefits across all the projects that get funded.) It is logical to ask this given that cost-effectiveness is closely related to the BCR (it has the same structure – it’s just that benefits are measured differently), and we’ve seen in PD322, PD323 and PD324 that ranking projects by BCR works in some situations but not others.
It turns out that the circumstances where it is logical to use cost-effectiveness to rank projects are equivalent to the circumstances where it is logical to rank projects using BCR.
(i) If you are ranking separate, unrelated projects, doing so on the basis of cost-effectiveness is appropriate. Ranking projects by cost-effectiveness implies that there is a limited budget available and you are aiming to allocate it to the best projects.
(ii) If you are ranking mutually exclusive projects (e.g. different versions of the same project), ranking on the basis of cost-effectiveness can be highly misleading. If there are increasing marginal costs and/or decreasing marginal benefits (which are normal), ranking by cost-effectiveness will bias you towards smaller project versions. In PD323, I said to rank such projects by NPV and choose the highest NPV you can afford with the available budget. If we are not monetising the benefits, there is no equivalent to the NPV — you cannot subtract the costs from a non-monetary version of the benefits. This means that, strictly speaking, you cannot rank projects in this situation (mutually exclusive projects) without monetising the benefits. If you absolutely will not or cannot monetise the benefits, what I suggest you do instead is identify the set of project versions that can be afforded with the available budget, and choose the project version from that set that has the highest value for the benefit metric. (Theoretically it should be the project version with the greatest net benefit (benefits – costs) but that is not an option here because in Cost-Effectiveness Analysis the benefits and costs are measured in different units.)
You don’t divide by the costs, but you do use the costs to determine which project versions you can afford. This is a fudge that only makes sense if you adopt the unrealistic assumption that any unspent money will not be available to spend on anything else, but it seems to me to be the best way to go, if monetising the benefits is not an option.
(iii) If you are ranking separate, unrelated projects, and there are multiple versions available for at least one of those projects, then cost-effectiveness does not work and the rule about choosing the highest-value benefit metric does not work either. Instead, you should build an integer programming model to simultaneously weigh up both problems: which project(s) and which project version(s). There is a brief video showing you how to do this in Excel in PD324. In the video, the benefits are measured in monetary terms, but the approach will work if you use non-monetary measures of the benefits.
There are a number of tools available for ranking projects based on cost-effectiveness (e.g. Joseph et al. 2009) but it is important to be clear that the approach only works in certain cases.
Even if you are using cost-effectiveness in the right circumstances (case (i) above), it has a couple of limitations relative to using BCR. One is that you cannot use it to rank projects with distinctly different types of benefits that cannot all be measured with the same metric. Another limitation is that cost-effectiveness provides no evidence about whether any of the projects would generate sufficient benefits to outweigh its costs.
Joseph, L.N., Maloney, R.F. and Possingham, H.P. (2009). Optimal allocation of resources among threatened species: a project prioritization protocol. Conservation Biology, 23, 328-338. Journal web site