The most startling aspect of the election for Mayor of London last week was that, though Sadiq Khan won, it was by a much narrower margin than the opinion polls had suggested. Had the polls proved accurate then Shaun Bailey, the Conservative candidate, would have been blamed.

So given that he performed so much better, it is right that this personal achievement should be acknowledged. In the days before the election, I was doing some phone canvassing for him and got a strong sense of a discrepancy from those expecting a Khan landslide. As I was ringing landline numbers it was skewed towards pensioners, who are much more likely to be Conservative supporters. But it was easy to find “switchers” of all ages – those who voted for Khan last time but would do so for Bailey this time. While the media were sneering, many voters felt Bailey cared about issues such as knife crime and had the experience and determination to tackle such problems.

The final result showed Khan beat Bailey by 55.2 per cent to 44.8 per cent – after second preferences had been included. Last time round, in 2016, Khan beat Goldsmith by 56.8 per cent to 43.2 per cent. So the gap narrowed from 13.6 per cent to 10.4 per cent. On first preferences, the gap narrowed from 9.2 per cent last time to 4.7 per cent. If Khan had widened the gap – merely by not as much as the polls had suggested – that would scarcely be much comfort for the Conservatives. But we saw a significant narrowing which does offer some sign of hope.

Naturally, there were special factors. There always are. Even many Labour supporters are unimpressed by Khan’s record of Mayor. Gimmicks and photo-opportunities. No significant achievements – least of all delivering what he promised. His personality has also become less attractive. When he was standing in 2016 he conveyed a great sense of energy – compared to Goldsmith who came across as rather languid in media interviews. But during this campaign, Khan was arrogant and petulant. Khan has been dismissive of his opponents and has not relished scrutiny or debate – which he would if he had confidence in his record. Some who would usually vote Labour felt uninspired and abstained.

All that dire opinion polling must have harmed Bailey’s campaign. Morale is important. If a candidate is perceived to have no chance then why donate to the campaign? Why volunteer? Why even bother to vote for him? Journalists had no difficulty getting off the record quotes from Conservative “insiders” on how they had given up; making Bailey the scapegoat. Melanie McDonagh in The Spectator wonders if Bailey would have won if he had not been undermined. Certainly, the treatment was unedifying.

So far as the London Assembly is concerned, no constituencies changed hands. But if we look at the “top-up” vote share, the gap also narrowed. Labour’s lead was 7.4 per cent, Last time it was 11.1 per cent. Labour are one seat down, the Conservatives one seat up. The demise of UKIP – who won two seats last time – may have helped.

Those figures tend to suggest that Bailey outperformed the brand. There were some Conservatives who gave anonymous suggestions that he should be deselected and a “stronger” candidate chosen. Usually, they didn’t come up with any names – certainly not of anyone who was willing to put themselves forward. I also doubt that if CCHQ had backed Bailey with more resources it would have been enough to clinch victory. A ten point gap is still significant.

The reality is that Bailey was the best candidate available. That does not mean I am uncritical of him. In Mayoral contests, voters are looking for some independence – even from candidates standing under a Party label. Bailey should have been bolder, less constrained about being “on message”, more radical in his policy offers. There should have been a greater willingness to take risks – he was always going to be denigrated and misrepresented by the Labour machine anyway. So he could have done even better than he did. But he did well and it is hard to see how he could ever have won.

What of the broader context? The 2019 General Election did not give a clear shift  in London. The Conservatives stayed on 21 seats – gaining Kensington from Labour but losing Putney to them; losing Richmond to the Lib Dems, but taking Carshalton and Wallington from them. Labour’s vote share was 48.1 per cent, down 6.4 percentage points on 2017. The Conservative’s share was 32 per cent, down by just 1.1 per cent. So on that measure the gap narrowed. But relative to the rest of country, the Conservatives in London fell further behind. That saw Labour’s vote share down 7.8 per cent while the Conservative share went up 1.2 per cent. The anti Brexit message from Labour and the Lib Dems went down better in London than elsewhere.

Next year we see the elections in London for the 32 borough councils. Last time round – in 2018 – these went pretty badly. Labour was ahead by 15.1 per cent in vote share. The Conservatives made a net loss of 92 councillors. Usually, we can get a clue of how things are going from local by-elections. Due to the pandemic, these have not been taken place. But we had a glut of them last week to catch up. 46 contests.

They were generally encouraging. There were Conservative gains in Enfield and Barnet – with no Conservative losses. In terms of swing, including the seats that did not change hands, there is an analysis on the Vote UK Forum. 11 seats showed swings from Conservative to Labour – all rather modest. Not only did far more seats swing in the other direction but much sharper. Hounslow, Greenwich, Croydon and Redbridge all in double figures. The contest in East Ham Central Ward on Newham Council saw a 21.1 per cent swing to the Conservatives. Thames Ward in Barking and Dagenham achieved a 26.2 per cent swing to the Conservatives. Both seats were easily held by Labour – which shows just how big Labour’s majority was last time. At present, neither borough has a single Conservative councillor. Could that change?

Other results in Lewisham and Islington showed the Green Party doing well. This may put pressure on Labour in wards where they have had big majorities in the past and so indirectly help the Conservatives to, at least, be competitive.

The fundamentals though remain challenging. Conservatives in the capital should not be relying on Khan to do an even worse job in his second term than his first. Nor should hopes of Conservative salvation rest on voters switching to the Green Party due to Labour and the Lib Dems not being regarded as left wing enough.

We need a lot more housing. Not more tower blocks. But beautiful new homes. Those with the ambition to own their own property or to start a family – or both – should not be forced out. Building on the scale required can be achieved. The frequently stated objection that there is “no room” in London is not correct. The planning reforms should help. Far more could be done to force the sale of surplus public sector land. Herbert Morrison was accused of saying that he aimed to “build the Tories out of London” – that was when he was Leader of the London County Council and hoped that the municipal estates being established would prove to be socialist fortresses.

The mission now is to build the Tories back in London. That means a great neo-classical revival. Setting locally popular design codes, releasing the land, and then allowing the developers to get on with it. It is a national priority. But nowhere is the need greater than in London.