Sir Julian Brazier is a former Defence Minister, and was MP for Canterbury from 1987-2017.

In the background to the unhappy struggles in the Conservative Party today is a philosophical clash in which the voices of libertarians are loudest. While (mostly) still supporting the man, their accusation is that the Johnson government has abandoned liberty.

These voices call for much that traditional “small c” conservatives should agree with – a smaller state, lower taxes, less regulation – but their message carries at its heart a deeply unhelpful strand which would be bad for the country, and calamitous for the Party’s prospects of staying in power.

Our most important domestic challenge today is reining back public expenditure so we can lower taxes on struggling families. Government spending is the highest proportion of GDP since the aftermath of the Second World War.

Where I part company from my libertarian friends is that I believe it is time we acknowledged that one of the hidden drivers of runaway public spending is libertarianism itself and its left-wing cousin, the human rights lobby. Both stress freedom and gloss over the responsibilities and consequences which should come with it.

John Stuart Mill formulated the paradox of hedonism: “those only are happy … who have their minds fixed on some object other than their own happiness.”

Similarly, the paradox of liberty is that we can only attain true freedom and a smaller state, if we focus not on selfish individualism but instead on nurturing and rebuilding those natural structures and attitudes which reduce the need for the services of the state. This requires active citizens, robust families, stronger communities and a sense of nationhood. These were themes of the late, great, Sir Roger Scruton.

One of his favourite examples were the American laws which allowed people, in most places, to build freely where they wanted, but then required the American taxpayer to expend huge sums taking roads and power to them. This has created a nightmare of ever-expanding suburbs with social black holes in town centres – and heavy government spending.

More broadly, he attacked the growing wish for extending freedoms without accepting any corresponding responsibilities, even – crucially – where there are heavy costs to the taxpayer and wider community (including later generations).

There is a parallel with Britain’s NHS. The cost of NHS and social care has exploded to the point where some are claiming Britain is becoming a health and social care system with a country attached. The Party is buzzing with ideas for reform of the NHS and social care – from pruning expensive bureaucrats and tackling GP contracts, to moving towards an insurance-based system. Yet there is one way we could reduce NHS spending dramatically and improve productivity in the economy: by persuading millions of obese people to lose weight and the nation to become fitter.

Scandinavian countries adopted a wide range of contrasting approaches to Covid but, with their much fitter populations, all suffered far lower rates of Covid deaths, – and lower pressures on their health systems. Indeed, the Swedish approach was never an option here because our large population of obese people would have brought the NHS down.

The impact of Britain’s obesity, the worst in Europe (apart from Malta), goes far beyond Covid. A range of illnesses from cardiovascular conditions to arthritis to diabetes are made both more likely and more dangerous by obesity – and also drive up the cost of the NHS.

Yet libertarians oppose measures to incentivise fitness, from sugar taxes to public health campaigns (what they call the Nanny State). Meanwhile the human rights lobby screams against ‘fat-shaming’ even in professions (such as the Army and the Police) where fitness is self-evidently important.

So, yes to lower taxation in general. But yes also to taxes targeting unhealthy foods – and to tax breaks for gym subscriptions.

A parallel example is opposition to so-called Covid passports. Most of the Covid deaths, for some time now, have been among the unvaccinated. All Conservatives should wish to raise restrictions as quickly as possible. Indeed, the noisy lobby calling until recently for the re-imposition of Covid restrictions was mostly on the Left, but the circumstances which have underpinned their case – the existence of large numbers who refuse to vaccinate and get sick – is ignored by libertarians and the human rights lobby.

By contrast, millions of Britons saw nothing wrong with those who choose to be refuseniks paying some price (in terms of minor inconvenience) for their potential impact on the NHS. Even as we manage to ease out of the last parts of lockdown, protecting the short-term liberties of the refusenik minority has consequences, not just for public spending, but also for many who have other life-threatening conditions over which – unlike the refuseniks – they have no choice. Sick refuseniks are occupying beds desperately needed by other sick people.

A broader example is attitude to the family. Individualists on left and right campaigned successfully a generation ago for the virtual end to restrictions on divorce and the end of allocation of fault as a factor in child custody and the division of assets.

Today, attempts to reinforce traditional families are bitterly opposed by many the same people. Iain Duncan-Smith’s radical reforms on social welfare reintroduced incentives to work, but he was consistently blocked in trying to remove disincentives for traditional families to stay together.

Yet the result of the decline of the traditional family is not just growing misery among children, with mental health, suicide, self-harm and drug-taking all on the rise – and mostly higher than other European countries. It is also extremely expensive for the taxpayer as social security spending and the requirement for police officers, social workers, prison officers and children’s mental health staff grows. Studies consistently show that stable two parent families offer – on average – the best outcomes for children and family breakdown has an immediate cost to the benefit system.

If the state can encourage responsible personal choices and the rebuilding of those Burkean structures, from the family to the community to a sense of shared nationhood, expenditure can fall as the use of the safety net declines. If, on the other hand, choices which lead to mounting bills for the taxpayer are protected on the basis that “We are not a country which asks to see papers”, the size of the state will expand as the safety net gets more and more crowded.

Scruton once commented “When government creates an unaccountable class it exceeds its remit, by undermining the relation on which its own legitimacy depends.” In courser terms, people hate a freeloaders’ charter; rights should be balanced by responsibilities.

Boris Johnson led us out of the European Union. The next moves we take should seek to re-establish that balance. So, yes to reducing regulation (such as the Clinical Trials Directive which destroyed East Kent’s biggest employer). Yes to making strategic choices to cut public spending and taxation (a smaller university sector, an end to the triple lock for pensions?). Yes to forging new global trade and wider partnerships.

But let’s have an end to the suggestion by so many of the Prime Minister’s critics that a combination of offering freedom, alongside state-funded protection from the consequences, will capture the hearts of the British people.