In the summer of 2004, when I began writing a life of Boris Johnson, reputable judges predicted he would be the next Prime Minister. It was 12 years since the Conservatives had won a general election, and ten years since Tony Blair became Labour leader.

Four Conservative leaders – John Major, William Hague, Iain Duncan Smith and Michael Howard – had failed to inflict lasting damage on Blair.

Perhaps it was time to send for Johnson, who possessed the curious attribute, for a scholar of Eton and Balliol, of being a celebrity. As Michael Gove suggested, in a defence of Johnson published in October 2004:

“Boris himself seems to recognise, whether intuitively or through observation, that celebrity now plays the role in politics that possession of an aristocratic name or a distinguished war record used to perform. It gives you a right to be heard. Celebrity allows you into people’s lives and homes, where they give you permission to share your views, in a way that others are denied.”

The celebrity living a life above the rules, appealing to our fantasies of effortless success, is a vulgar figure. But it seemed to me that there was more to Johnson than that, a curious amalgam of humanity, selfishness, toughness, vulnerability, loyalty, unreliability, flexibility, stubbornness, energy, laziness, wit, recklessness, ambition, linguistic invention and the ability to transform the atmosphere on entering a dull shopping centre.

He entered the Commons in 2001, has never devoted as much time as he should have done to becoming a skilful parliamentarian, and in the summer of 2004 was somehow managing, in defiance of prudent advice, and of assurances he had given, to edit The Spectator while playing a shadow ministerial role.

I had known him, though not particularly well, since 1987, and I thought, regardless of whether predictions of a brilliant future were fulfilled, he would make an enjoyable subject for a biography.

When I told him of my idea, he laughed for a long time before saying: “Such is my colossal vanity that I have no intention of trying to forbid you.”

But very soon he began to get cold feet, and after a few months he attempted to discover how large my advance was, and to buy me out of writing the book.

“If it’s a piss-take that’s OK,” he said at an early stage, but went on: “Anything that purported to tell the truth really would be intolerable.”

I contended that on the contrary, politicians almost always get into trouble, not for telling the truth, but for trying to conceal it. Painful episodes in his past would lose their power to hurt him once they were known, and it would be much less dangerous to deal with this stuff now than when he became Prime Minister.

In late 2004, Johnson suffered a number of setbacks – the forced apology to the people of Liverpool, the “inverted pyramid of piffle” affair, his sacking by Michael Howard from the Opposition front bench – so severe they would have driven a less resilient figure out of politics.

I am confident he never for one moment considered retiring from the fray and prostituting his talents as a vacuous, over-paid television presenter.

But in 2005, when Howard stepped down after leading the party to its third defeat in a row, there was clearly no possibility of Johnson becoming a Conservative leadership contender. He instead backed David Cameron, who came through and won.

The first edition of my book appeared in 2006, and made no difference to anything. Johnson wrote in the copies people asked him to sign “it’s all rubbish” and other messages which amused the recipients.

The new leader, who almost until taking office had been much less well known than Johnson, proceeded to keep him at a distance. In the Shadow Cabinet reshuffle of July 2007, Cameron promoted Gove, who had only entered Parliament in 2005, to the key role of shadowing Ed Balls, whom Gordon Brown had promoted to the post of Secretary of State for Children, Families and Schools.

Johnson was left in the post he already had as Shadow Spokesman on Higher Education. It had become clear that as long as Cameron was leader, Johnson was not going to get anywhere at Westminster.

So after much agonising, he took the risk of becoming the Conservative candidate who in 2008 would take on Ken Livingstone, the Mayor of London, at this point reckoned to be pretty much invincible.

After several months during which, at Lynton Crosby’s urging, Johnson told no jokes, he defeated Livingstone and entered City Hall. In 2012, he beat Livingstone again, and in 2015, not long before the end of his second term as mayor, he re-entered the Commons at the general election in which Cameron gained – to the commentariat’s surprise – an overall majority.

Cameron had promised that if he won, he would hold an EU referendum. Here was another decision for Johnson. Could he bear to become a subordinate cog in the Cameron-Osborne machine?

He could not. After more agonising – for he possesses a well-hidden streak of prudence – he became the most prominent figure in the Leave campaign, to which Gove had lent a degree of intellectual respectability.

The commentariat believed Remain would win, but turned out not to know its own country as well as it thought it did. On 23rd June 2016 Johnson led Leave to victory.

Cameron, after nearly 11 years as party leader, at once resigned, and for a few days Johnson was the favourite to succeed him. But at 9.02 a.m. on the Thursday after the referendum, Gove sent an email to journalists which said:

“I have come reluctantly to the conclusion that Boris cannot provide the leadership or build the team for the task ahead. I have, therefore, decided to put my name forward for the leadership.”

Johnson withdrew from the race, and Theresa May, who looked the only grown-up left, became Prime Minister. She proceeded to make Johnson Foreign Secretary. These convulsions I describe in the most recent edition of my book.

But she never used Johnson, or brought him properly into her team. Successful Prime Ministers have usually travelled to major international events with the Foreign Secretary they appointed.

That was not May’s way. She created a new Brexit Department, supposedly to deal with the great issue arising out of the referendum, but that too played little part.

In a manner that recalled Tony Blair, Anthony Eden and Neville Chamberlain at their least successful, she set out to run foreign policy from Number Ten.

Johnson won poor reviews as Foreign Secretary, and not much better after his resignation in July 2018 in protest at the Chequers plan for Brexit. The commentariat reckoned he was probably out of it.

As so often, it was wrong. Johnson has run a more professional campaign than on previous occasions, while his rivals could not decide which of them would run against him, and have instead run against each other.

I confess to feelings of bemusement, even incredulity, at this turn of events. People ask me “when is your new edition out?”, but for the next few weeks, it would seem both rash and impertinent to assume Johnson will definitely become the next occupant of Number Ten.

Conservative members will vote as they wish. So far as I know, nothing I have ever written or said about Johnson has changed a single mind. All I may occasionally have done is provide some of the evidence, both for and against, needed to support a view already arrived at instinctively.

For Johnson dramatises the temperamental divide which runs through our history and our nation

“between the Cavaliers (Wrong but Wromantic) and the Roundheads (Right but Repulsive).”

Having just glanced at1066 And All That in order to check the quotation, I cannot resist quoting the next paragraph, with italics as in the original:

“Charles I was a Cavalier King and therefore had a small pointed beard, long flowing curls, a large, flat, flowing hat and gay attire. The Roundheads, on the other hand, were clean-shaven and wore tall, conical hats, white ties and sombre garments. Under these circumstances a Civil War was inevitable.”

A sort of Civil War rages now. Johnson the Cavalier brings out the puritanical instincts which lie buried in such otherwise genial figures as Matthew Parris, Max Hastings, Simon Heffer and Bruce Anderson.

They simply cannot bear his style of politics, consider him incapable of learning from the mistakes and misjudgments he has made, and condemn him in unmeasured terms. There can be no modern Conservative who has been denounced by a larger number of well-known pundits.

A considerable number of MPs of all parties have similar feelings about him. They stiffen when you mention him. But others in increasing numbers think Johnson is now needed, and that he has developed the qualities needed to run a brilliant team.

It is all rather bewildering. Never have a received a greater number of requests for interviews about Johnson, and to appear on various programmes along with his other, admirably unimpressed biographer, Sonia Purnell. The Americans, the Germans and the Japanese all want to know more about him.

And the more he annoys the Establishment, the better he appears to do. One might almost think his time has come.