Nicholas Boys Smith is the director of Create Streets and was co-chair of the Building Better Building Beautiful Commission with the late Sir Roger Scruton.

If you had forgotten that planning is the toxic third rail of British politics, the Chesham and Amersham by-election result last week will have reminded you. The facts are stark. A Conservative majority of 16,000 slain and reshaped overnight into a Liberal Democrat majority of 8,000. Liberal Democrat leaflets artfully evoked middle England’s worst fears: “automatic planning permission granted,“ “power handed to developers to build on green spaces,“ “residents’ right to oppose developments removed” – and “The Chilterns must be protected.” Incendiary quotations from the Campaign to Protect Rural England. Journalists opining that these messages “cut through.” Former Conservative leaders Theresa May and Iain Duncan Smith opposing the plans and Isle of Wight MP, Bob Seely, citing “significant pushback from communities on planning.”

So is that it? Is that the end of planning reform for another generation? I hope not. Because, despite the good intentions of many of its practitioners, the British planning and development process is one of the most complex and expensive, risky and regressive, anywhere in the world. Since 1947, the right to develop in the UK has been nationalised. But the implementation of that nationalised planning right is profoundly unpredictable. A new building in England needs a planning permission; a case-by-case judgement by a planning officer. This judgement is based on the local plan which is a policy document, not a regulatory one. It gives principles and guidance. It doesn’t set rules. Knowing what you can build, “winning” permission (a telling phrase) takes time, judgement, experience – and lots of money.

This is fundamentally different to most other countries where the right to develop is not nationalised but regulated. In countries as diverse as America, France and Germany, as long as landowners follow the local regulations, the complexity and cost of development is very modest compared to the UK. In Germany, for example, the freedom to build is a part of the constitutionally-guaranteed definition of property. Hardly surprisingly, far more homes than pretty much everywhere else are built by small developers or self-builders.

In contrast, our complex and risky process has not just retarded the rate of housebuilding, but created a near cartel of the largest developers as well as a boondoggle for consultants and lawyers who make a good living feeding the current process a constant diet of expensive reports and assessments. This adds up to a savage trick played by the old upon the young. A smaller proportion of people born between 1981 and 2,000 are homeowners, at this life stage, than for any previous generation since 1926. And their rent payments have increased from ten per cent of net income 30 years ago to around 30 per cent now. This has enhanced generational inequality on a seismic scale with immense political ramifications. Britain’s housing challenges are not just retarding the age of home ownership. They are fundamentally changing generational fairness, particularly in the south-east. The sight of those who espouse progressive principles supporting a system which requires inherited wealth, large cash flows, or expensive lawyers, to create homes is, to put it politely, quite surprising.

It is a question of how and what as well as how many. Our over-reliance on a small number of big developers has consequences. When I co-chaired the Building Better Building Beautiful Commission with the late Sir Roger Scruton, one of the most consistent messages we heard from members of the public was that they felt “done to”, “built at”. One told us:

“My local experience is that the community is seen as an inconvenience to be swept aside during the planning process. Consultation has fallen to almost nil…. developers hold considerable sway.”

The evidence strongly suggests our correspondents were correct. Polling shows homes built by smaller firms tend to be better. And a study of every property sale in six British cities found a premium associated with older neighbourhoods up to seven times greater than the premium for new build homes. Two thirds of British adults say they would never consider buying a newly-built home. And, given their lamentable build quality, you can hardly blame them. A recent survey by UCL found that at least three quarters of new developments were mediocre or poor. Where are the architects you ask? Sadly, too many (not all) are dismissive of public preference (“the public need educating to understand good design” is one of the most noxious concepts) and the volume housebuilders simply don’t employ them in consequence. Everyone loses.

As the results in Buckinghamshire show, the knee jerk assumption in Britain over the last 70 years has therefore firmly, and rationally, become that new development will be bad and is best prevented by anyone who loves their local neighbourhood. Frankly, I don’t blame NIMBYs for not wanting most new homes anywhere near them.

So the current situation is indefensible. We’re not building enough homes. We’re not creating good enough new places. And confidence in the process is so low (two per cent for developers, seven per cent for planning) that all attempts to improve the situation are meet with knee jerk and politically exploitable resistance.

If the government wants to win through, they’ll have to frame the debate, and the reality of what they are proposing, in very broad terms. The answer is not just about numbers or planning but about the location, nature and quality of the new places we are creating and our stewardship of the old. We need to engender a renaissance of civic pride and revitalise the great tradition of civic involvement. Beautiful, popular, healthy and sustainable new places should become the natural result of working within the system, not the consequence of working against it.

I would say this wouldn’t I – but I think the recommendations of the 2020 Building Better Building Beautiful Commission can point the way. Here are four key themes.

One theme is indeed planning and communities. We need shorter, more visual local plans, setting a predictable level playing field so that we cease to overly rely on the big boys. These shorter more visual plans should be very provably linked to what people locally like and prefer, making use of dramatically improving digital engagement and visual preference surveys.

Planning must aim to create what local people like and believe to be homely and beautiful. Development should be a net gain not just “no net harm.” We need to bring the democracy forward so that more of it takes place in setting the local plan. We then need to create multiple locally-led fast tracks to beauty and to locally-improved places or new homes. One idea with potential is “street votes” – voluntary intensification of the suburbs on a street by street basis. There should also be even stronger support for community-led development.

A second theme is stewardship and tax. We need to incentivise responsibility to the future, not penalise it. We need to move from a “build by unit” model to a ‘patient capital’ model. UK tax codes encourage a short-term approach to development by often doubling the proportional tax bill to landowners who co-operate or maintain a long term interest. This is unintended but it is perverse. It should be changed. There needs to be a level tax field between different approaches not an incentive to take a bad approach. Industry bodies, landowners and government should co-operate to create a new recognised stewardship “kitemark” which should have a series of legal and management standards on the approach to land and development.

A third theme has only grown in salience since we published last year and, possibly changed its nature due to the revolution in online working. It is regeneration and sustainability. We must end the scandal of “left-behind” places. Too many places are losing their identity or falling into dereliction. They are noisy, dilapidated, polluted or ugly, often scarred by fast roads through what should be their thriving centres. Such places provably create fewer jobs, attract fewer new businesses and have less good schools. They do not flourish.

Government has committed to ending the scandal of “left-behind” places. Excellent. But it is never enough to invest in roads or shiny “big box” infrastructure. Development should be regenerative not parasitic. A member of Cabinet should be responsible for ensuring that new places reach the right standards, co-ordinating perspectives between the ‘triangle’ of housing, nature and infrastructure.

At the local council level, there should be a Chief Placemaker in every senior team and a member of the local Cabinet who has responsibility for placemaking. Government should align VAT on housing renovation and repair with new build, in order to stop disincentivising the re-use of existing buildings. The built environment sector is currently responsible for 35-40 per cent of total greenhouse gas emissions in the UK. A new-build two bedroom house uses up the equivalent of 80 tonnes of CO2. Refurbishment uses eight tonnes. Even with the highest energy-efficient specification, new build would take over 100 years to catch up. The embodied energy in the bricks of a typical Victorian terraced house would drive a car more than ten times around the world. The greenest building is the one that is already built. Beautiful buildings are conserved and adapted. Ugly buildings are torn down and replaced.

A fourth theme is management and the way the public sector intervenes in our towns and streets. Too often over the last 70 years, the public sector has made places uglier and less prosperous. We need to change corporate performance targets for highways, housing and planning teams in public bodies. They should be targeted on objective measures for well-being, public health, nature recovery and beauty (measured inter alia via popular support). We should be measuring quality and outcomes as well as quantity. There is an urgent need to improve the procurement targets, process and scoring within central and local government and, Homes England. Finally, new public buildings should be popular and beautiful sources of civic pride with polling on local popular design preferences as a normal part of the procurement process.

This all amounts to a profound generational change from a vicious circle of parasitic development to a virtuous circle of regenerative development. As the recent by-election results show, trust in new places will not be reborn in one parliament. It will take many years. (And I have not even had space to touch on the need to re-green our streets with millions of street trees or improve architectural and planners’ education to be more practical with more focus on popular preferences and the associations of urban form and design with well-being and health).

The good news is that recent changes to what is called the National Planning Policy Framework following from our report are starting to move the dial in the right direction. This important overarching document now asks for beauty more clearly, makes it easier to refuse ugliness and expects a biodiversity ‘net gain’ on sites. At Create Streets we are starting to see this make a change, for the better, on the ground. But there is much more to do. And a wide discussion to have over a generation.

There is no fundamental reason of economics or technology that prevents us creating streets and squares, homes and places of humanity and beauty, places in which we can lead happy, healthy and connected lives, know more of our neighbours, and be more joyful as we go about our daily life. We, as a society, have just not done it and we are paying the consequences. Roger Scruton wrote that:

“Home is not occupied only by us: it is inhabited by the ghosts of our ancestors, and by the premonition of children who are yet to be. Its essence is continuity, and it provides the archetype of every experience of peace.”

I hope that, 50 years hence, more of our fellow citizens will be ‘living with beauty’ and leading happier, healthier, more sustainable and better-connected lives in consequence. To achieve this will require planning reform. But it will require far more. We have started the journey: neighbourhood groups planting street trees, new local community land trusts building local homes, the government facing into the wild winds of planning reform. I pray that, as a society, we are able to finish it.