Ryan Bourne occupies the R Evan Scharf Chair for the Public Understanding of Economics at Cato, and is the author of Economics In One Virus.

A year since the first UK lockdown, the economic question most asked about Covid-19 is whether shutdowns would pass a simple cost-benefit test.

Such analyses have been demanded by Covid Recovery Group MPs and, indeed, the correlation between those wanting such evaluations and lockdown scepticism is now strong. Philip Lemoine, a shutdown critic,even suggested in the Wall Street Journal recently that the absence of government economic appraisals worldwide showed policymakers knew lockdowns would fail these economic tests.

I think there’s a more charitable explanation to policymakers’ reluctance to do them: a lockdown cost-benefit analysis is actually incredibly difficult to do well.

Certain issues, such as how much we should value lives saved from mitigating death risks, are contentious at the best of times. But the costs and benefits of crude business shutdowns, school closures, and stay-at-home mandates are far more uncertain than, say, assessing a minor work safety regulation.

First, defining the counterfactual from which to measure lockdowns’ marginal impacts is hard. Clearly, the alternative to lockdowns is not an unmitigated spread of the virus. Evidence from around the world shows that countries which spurned lockdowns also saw waves of infections, rather than a massive outbreak leading to herd immunity. Voluntary social distancing and other mitigation attempts, in other words, appear sensitive to the prevalence of the disease. When cases get high enough, human contact falls, eventually pushing the transmission rate of the virus below one.

The problem is that when these tipping points occur appears a moveable feast across time and countries, unmoored from any consistent prevalence threshold, influenced by pandemic fatigue, and probably strongly affected by chatter of potential lockdowns too. That leaves scope for lockdowns reducing human contact to have big marginal benefits if applied earlier, or in accelerating the downswing of curves. But it also makes it incredibly difficult to precisely set out what would happen in a counterfactual world.

Sceptics are right that unpicking lockdowns’ precise impact requires more than just eyeballing curves. Particularly because a government using lockdowns might exacerbate waves: if people expect governments to lockdown when things are bad, they might begin judging unlocked periods as “safe.” But it would be laughably convenient if you assumed the public would voluntarily mimic the effects of lockdowns precisely when they would otherwise be introduced, as some UK sceptics appear to allege.

A second, related difficulty in lockdown appraisal is that the costs and benefits change with our medical capabilities and knowledge of the virus: they are time and context specific. The UK’s first lockdown was introduced in spring 2020 under a cloud of uncertainty, with fears that, absent evasive action, hospital capacity could be overwhelmed, bringing severe social costs. At that time, we had little firm knowledge about whether a working vaccine would materialise either, and so whether “deaths averted” by lockdowns were merely “deaths delayed.”

The current national lockdown, in contrast, was introduced because of the highly transmissible variant and then-imminent vaccine rollout. The short-term benefits of this lockdown were thus more certain. But as vulnerable groups have now been jabbed at least once, the marginal benefits of lockdown days have now fallen dramatically, while the marginal costs of lockdowns continue to rise. Accounting for changing dynamics within a cost-benefit analysis, influenced by the availability of testing or vaccines that help “lock-in” any suppression of the virus, is therefore crucial to understanding lockdowns’ net effect.

A third and final difficulty arises because a whole range of lockdowns’ costs and benefits are subjective or highly uncertain. The worst attempted analyses so far just calculate the estimated value of lives saved and compare those to some estimate of lockdowns’ impact on GDP or, worse, government spending.

But this is hugely incomplete, ignoring worst-case tail risks and a range of other obvious impacts. On lockdowns’ benefits, for example, a lot of people self-evidently value avoiding infection risks, not just death risks. Some economists have even calculated that mitigating such non-fatality risks might double the monetary value of lockdowns’ health benefits.

On the cost side, the uncertainties are legion. Lost economic output is clearly a mere subset of lost wellbeing. How much should we value losses stemming from, say, someone being unable to attend the wedding of a family member, or visit a friend with depression? These are extremely subjective, but accumulate to very large costs indeed. That’s before we consider lost schooling’s effects on children’s life chances, or attempt to account for the impact of government shutdowns on entrepreneurialism.

No cost-benefit analysis I’ve seen so far has gotten close to developing a robust, country-specific counterfactual to lockdowns, nor accounted for all these effects. But that it is difficult doesn’t mean governments shouldn’t have attempted it, or that we now shouldn’t endeavour to carefully and retrospectively deliver such analysis.

Some dismiss cost-benefit analysis of lockdowns on the grounds that they may mislead the public given these uncertainties. Others, such as Mark Carney, think weighing health benefits against lost economic and social liberties is alien to society’s preferred values. The public clearly wanted to minimise the health impacts of Covid-19, Carney says, so governments should seek to deliver that in the least costly way possible.

Carney may be right on the psephology of what voters wanted and how this drove decision-making. But the trade-offs were real, and analysts shouldn’t shy away from highlighting them. Indeed, my own views on the UK lockdown experience are nuanced. They clearly “worked” in the sense of significantly reducing cases and deaths each time, while the case for them was strongest in Spring 2020 and December 2020, given the contexts of uncertainty and vaccine imminence.

And yet, overall, I still think the re-use of lockdowns after the summer Covid-19 lull is evidence of the failure of the government to think on the right margins. Even if lockdowns passed cost-benefit tests at certain times, that doesn’t make them “optimal.” The inability of the government to harness new knowledge and to unbundle the aspects of lockdowns that clearly imposed costs for scant public health gains, even after months to devise something less painful, is striking. Other countries likewise show how testing, tracing, cluster-busting, and guidance on ventilation, if done well, could have locked-in a lot of the gains of suppression without the massive downsides.

During the next few years, careful analyses will seek to unpick the effects of lockdown relative to counterfactual worlds. Such are the contestable assumptions and uncertainties, though, I suspect we’ll never get a cost-benefit analysis that “resolves” this debate.