Every 20 or 30 years, the Conservative Party stages a leadership contest which affords huge enjoyment to those who love political theatre.

Tory MPs who in normal times suffer from a painful sense of personal insignificance feel their spirits rise during these spectacles, for at long last they know their votes matter, and can hope the victor will recognise their hitherto disregarded abilities.

Ambitions which have lain prudently suppressed for decades burst out into the open as the sudden death tradition of British and especially of Conservative politics reasserts itself.

Over the last century, such exceptional contests have occurred in 1922, 1940, 1963, 1990 and 2016, so we should not be due another one for quite some time.

But the restlessness which precedes an eruption is now perceptible: the feeling that however dangerous it may be to overthrow the leader in favour of some untried replacement, not to act would be still more dangerous.

And it is possible that whatever happens in the near future may come to be seen as a powerful aftershock from the 2016 contest, rather than a new event.

It is the personal element in these fights which rivets the spectators’ attention. We witness the downfall of a leader who seemed impregnable, and the rise of someone else who seemed out of the question.

Yet it would be an error to regard these as purely personal battles. For as Jacob Rees-Mogg remarked a few days ago, “The leader is important, but the party is more important.”

Such leadership contests are in essence about bringing the Conservative Party into closer alignment with what the British people want. They are part of the instinct for survival which has enabled the party to outlive and often outperform its competitors ever since its foundation by Sir Robert Peel in the 1830s.

Over that period, only two Conservative leaders, Lord Salisbury in 1902 and Stanley Baldwin in 1937, have left of their own volition.

Let us work backward from 2016. In that year, the EU referendum was held, David Cameron lost, and therefore he had to go, as he himself recognised, with his usual quick-wittedness, by breakfast the morning after.

The nation demanded someone who would implement Brexit, which by definition could not be Cameron, for he had insisted Brexit would be a disaster. Boris Johnson, who had helped swing the referendum against Cameron by insisting Brexit would be fine, was to general astonishment knocked out of the race, because his ally, Michael Gove, declared him unfit to be Prime Minister.

Theresa May came through as the unity candidate, who had been a tepid Remainer, but was now, she declared, a convinced Brexiteer. She is in trouble because fewer and fewer Brexiteers believe this to be true, and because she called a general election at which she failed to liberate herself from the stern, unbending Brexiteers.

Who will replace her? That is for the party to decide, while striving to interpret what the British people want. In my opinion, what the people want is a Brexit conducted with brio, which is what Johnson offers. But we shall see.

By 1990, Margaret Thatcher had exhausted the patience even of such titans as Sir Geoffrey Howe and Nigel Lawson, who had been among her staunchest supporters. On the poll tax, and on Europe, many of her Cabinet colleagues and her backbenchers now thought she was leading them to perdition, so out she went.

She was succeeded by John Major, who had the merit of not being Michael Heseltine, who had raised the standard of rebellion against her. Tory leadership contests are highly unpredictable, and the front runner has not won since 1955, when Anthony Eden, loyal crown prince since 1942, succeeded Winston Churchill.

In 1963, Lord Home succeeded Harold Macmillan, after bizarre scenes at the Tory Party Conference in Blackpool. At the start of that week, Home did not even appear to be a contender.

The press believed Rab Butler would win, but he lacked the killer instinct, and Macmillan was determined not to have him. Of the other contenders, Lord Hailsham ruled himself out by engaging in vulgar stunts, while Reginald Maudling gave a dull speech which prevented him from building up any kind of momentum.

Home’s conference speech began with the words, “I am prepared to offer a prize to any newspaperman this morning who can find a clue in my speech that this is Lord Home’s bid to take over the leadership of the Conservative Party.”

He backed his way skilfully into the limelight. It was his natural game, and as a man of unimpeachable integrity, he cleared the air after the Profumo scandal. But Home had great difficulty dealing with the Labour leader, Harold Wilson, and failed by four seats to win the 1964 election, so did not become an example of the Conservatives regaining the initiative.

In 1940, Winston Churchill took over from Neville Chamberlain, for the country needed a fighter, and a man capable of bringing the Labour Party into government, rather than an appeaser, and a man who since the 1920s had viewed Labour with scorn.

But since the Conservative Party at this point distrusted Churchill, and was for the most part loyal to Chamberlain, this astonishing episode cannot be taken as proof of Tory perceptiveness.

In 1922, Bonar Law had temporarily resigned the leadership of the Conservative Party owing to ill-health, and Austen Chamberlain (older brother of Neville) had taken over, and wished to persuade the party to fight another general election under the dangerously autocratic and corrupt leadership of the great Liberal war leader David Lloyd George.

In October 1922, Chamberlain called a meeting of all Conservative MPs at the Carlton Club, to get them to support his plan of fighting another general election under Lloyd George’s leadership. Law proceeded to warn, with what Robin Harris describes as “low-key brutality”, that the needs of the party, not the coalition with the Liberals, must come first: “The party elects a leader, and that leader chooses the policy, and if the party does not like it, they have to get another leader.”

Chamberlain pushed the question of whether the Conservatives should continue to support Lloyd George to a vote, and lost by 187 to 88.

So Law succeeded Lloyd George as Prime Minister, and was himself succeeded (after falling mortally ill) by Baldwin, who dominated politics until 1937.

Law’s words at the Carlton Club endure to this day, and should be heeded by May: “The party elects a leader, and that leader chooses the policy, and if the party does not like it, they have to get another leader.”