The choice of the next Conservative leader, and Prime Minister, must be seen to be fair. Only an open and democratic contest, conducted according to equitable rules, will confer legitimacy on the victor.

But what does the word “democratic” mean in this context? A Conservative leader needs the support of three electorates whose priorities often conflict: the Party’s MPs, its members, and the wider voting public.

In 1963, Humphry Berkeley, a young Conservative MP, described the party’s then method of choosing its leader as “more appropriate for the enstoolment of an African chief”.

At that time, the new leader “emerged” following  “the customary processes of consultation” within the Tory tribe, an opaque process conducted by senior figures.

In 1957, when Sir Anthony Eden, his reputation shattered by Suez, resigned citing ill-health, Lord Salisbury (known as Bobbety, and unable to pronounce the letter “R”) and Lord Kilmuir interviewed members of the Cabinet to see whether their preferred successor was  Rab Butler, whom the press expected to win, or Harold Macmillan.

Kilmuir described in his memoirs how this was done:

“Bobbety and I asked our colleagues to see us one by one in Bobbety’s room in the Privy Council Offices, which could be reached without leaving the building. There were two light reliefs. Practically each one began by saying: ‘This is like coming to the Headmaster’s study.’ To each Bobbety said: ‘Well, which is it, Wab or Hawold?'”

It was by a large margin Harold Macmillan. In 1963, when he in turn, his reputation impaired by the Profumo affair, resigned citing ill-health, Rab Butler was again regarded as the favourite, and was again beaten, this time by someone not even thought to be in the race.

The victory of the 14th Earl of Home took almost everyone by surprise. He was able to disclaim his hereditary peerage thanks to recent legislation passed as a result of a campaign by a Labour MP, Antony Wedgwood Benn, who had become Viscount Stansgate on the death of his father but wanted to stay in the Commons.

Home, who had left the Commons in 1951 on inheriting his peerage, re-entered the House at a by-election as Sir Alec Douglas-Home. But some of Butler’s supporters were deeply upset by his defeat, and Iain Macleod wrote a celebrated and profoundly damaging article for The Spectator in which he suggested the leadership race had been fixed by “the magic circle” of senior Conservatives, eight out of nine of whom had been to Eton.

This was by no means the first time the Conservatives had felt a need to show themselves more democratic in their leadership procedures. As early as 1922, Bonar Law had refused, after the famous Carlton Club meeting at which Tory MPs voted to break up the coalition with Lloyd George, to accept the King’s invitation to form a Government until he had himself been re-elected as Conservative leader.

As Robin Harris writes, in The Conservatives: A History,

“The Conservative Party was not yet democratic in its procedures. But without adumbrating any new doctrine, the Carlton Club meeting had imposed a new quasi-democratic reality, one which no leader would be able to overlook. At a further party meeting held on Monday, 23rd October [four days after the Carlton Club meeting] at the Hotel Cecil, Law was proposed by Curzon, seconded by Baldwin, and duly elected by unanimity. Later that day he took office as Prime Minister.”

Law’s main need was to try to hold together a badly divided party, in part by showing it was united in accepting his leadership.

Home’s main need, four decades later, was to show that the Conservatives were adapting to the more democratic and classless atmosphere of the 1960s, when it was no longer tolerable for politicians educated at Eton (as Eden, Macmillan and Home all were) to appear to be sharing out the prime ministership among themselves, especially from January 1963, when Hugh Gaitskell, the Labour leader and a Wykehamist, unexpectedly died, and was succeeded by Harold Wilson, known already for his plain man act:

“The Right-wing Establishment has never tried to embrace me or buy me off. That’s probably a compliment. Lady Whatsit or Lord So-and-So haven’t plied me with invitations. I don’t do much socialising and my tastes are simple. If I had the choice between smoked salmon and tinned salmon I’d have it tinned. With vinegar. I prefer beer to champagne and if I get the chance to go home I have a North Country tea – without wine.”

After leading the Conservatives to a narrow defeat in the 1964 general election, which he reckoned he might have won if Macleod and another Butler supporter, Enoch Powell, had “pulled their weight”, Home set about the necessary process of democratisation. In his autobiography, The Way The Wind Blows, he wrote:

“After the widespread criticism of the methods which had been used in my own selection as Leader, I decided that for the sake of any future holder of the post, the processes of change must be reviewed.

“The ‘Magic Circle’ of selectors had almost everything to be said for it. The Whips and the experienced Conservative Parliamentarians knew the form of every runner in the field; they knew the Members of Parliament who had to work and live with the chosen Leader; and they could operate quickly and quietly in collecting views. It was the latter ‘advantage’ which caused the trouble. Some felt that candidates favourable to the establishment had the edge over anyone who might at any time have been rebellious, and there were always those who, stirred up by the media, were ready to charge the ‘Magic Circle’ with rigging the result.

“I was not particularly worried about such happenings until the accusation that the last result had been jobbed began to reverberate through the Constituencies and to affect Party morale outside Parliament. I then came to the conclusion that, with all its disadvantages, it was necessary to adopt a system of election of a leader, where from start to finish everything was seen to be open and above board. I was in the best position to see that business through.”

The rules decided upon by Home, with the help of Lord Blakenham, the Party Chairman, William Whitelaw, the Chief Whip, and Berkeley, who submitted a memorandum, were as follows.

The leader would be elected by Conservative MPs, with the process presided over by the Chairman of the 1922 Committee. Candidates needed a proposer and a seconder. If on the first ballot, a candidate received both an overall majority and 15 per cent more of the total number of votes cast than any other candidate, he would be elected.

If not, a second ballot would be held, with an overall majority sufficient for victory. If there was still no winner, the three candidates receiving the highest number of votes would go forward to the third and final ballot, in which each voter had to indicate two preferences, marking his paper “1” and “2”.

No provision was made for annual re-election. Home later said that “once a party had elected a leader that was that, and it had better stay with him”. He may also have assumed that any leader who became deeply unpopular with his own MPs would have the decency to step down.

These details are taken from A Conservative Coup: the Fall of Margaret Thatcher  by Alan Watkins. Soon after setting up the system, Home himself stepped down, and in 1965 the Conservatives held their first leadership election.

Reginald Maudling was the favourite, Edward Heath a strong challenger, and Powell ran as an outsider. The result of the first ballot was: Heath, 150; Maudling, 133; Powell, 15.

As Watkins observes, to satisfy the 15 per cent rule, Heath needed a majority of 45, whereas he only got one of 17. But Maudling and Powell now withdrew, leaving Heath the only candidate nominated for the second round, whereupon Sir William Anstruther-Gray, Chairman of the ’22, declared him the winner, a result confirmed at the ceremonial party meeting held six days later.

Heath was the Tory answer to Wilson, but here was an early sign that the rules could not cover all eventualities, including the resignation of candidates who saw they had no chance of winning. And Watkins makes another crucial point:

“The rules were not properly understood by many Conservative MPs, either in 1965 or on the three subsequent occasions on which a modified procedure was used. It was not merely that they tended to become glassy-eyed whenever percentages were mentioned: they were also what Anthony Crosland would have called ‘frivolous’ in their attitude to elections. They failed to understand that elections were a serious business which produced important results. They regarded them much as dissatisfied Conservative voters looked upon by-elections, as an opportunity to register a protest. Thus several Conservatives voted for Mr Heath, not because they wanted – still less because they expected – him to win, but because they wanted to administer a shock to ‘the old gang’ as represented by Maudling.”

By the end of 1974, when he led the party to two general election defeats, Heath was unpopular with many of his own MPs, whom he treated with quite remarkable rudeness. He was on very bad terms with Edward du Cann, the Chairman of the ’22, but in November acceded to his suggestion that there should be a review of the leadership rules.

Heath asked Home, who was by now once more a peer, to carry this out. In December, Home’s committee recommended that an annual election should, if requested, take place, and that the 15 per cent “surcharge” should be of all Tory MPs, not just of those who voted – a provision which made victory on the first ballot more difficult.

After much indecision among Heath’s opponents, Margaret Thatcher declared her willingness to stand. The assumption was that she would not win. On the first ballot she got 130; Heath, 119; Hugh Fraser, 16.

That was a tremendous shock. New candidates entered the second ballot, and the result was even more shocking: Thatcher, 146; Whitelaw, 79; Sir Geoffrey Howe, 19; Jim Prior, 19; John Peyton, 11. She had won an absolute majority of 18, and would remain leader for 15 years.

By 1989, she was unpopular, and Sir Anthony Meyer ran against her as a stalking horse. She got 314 votes to his 33, but 24 more MPs had spoiled their ballots, and three had abstained, so the total opposition to her was 60.

In November 1990, Michael Heseltine mounted his challenge. On the first ballot she obtained 204 votes to his 152, with 16 abstentions. She was four short of the 15 per cent “surcharge” of those entitled to vote.

This was a failure for Thatcher, who was quite soon persuaded that she would have to step down. But as soon as she had done that, things became very difficult for Heseltine, whose guns were trained on her, his whole campaign predicated on the assertion that he could save Tory MPs’ seats and she could not.

Nick Budgen, the MP for Wolverhampton South West, was keeping me in touch during this tumultuous period with the attitude to Heseltine of Thatcher’s supporters in the constituencies, and reported: “Their main feeling is stop that long-haired bastard. They don’t much care what animal they use to stop him.”

John Major and Douglas Hurd entered the second ballot, so there were two animals to choose from, of whom Major soon looked the better bet, though Budgen, with characteristic perversity, said he was supporting Hurd, as he would like to be betrayed in style.

The voting in the second ballot was Major, 185; Heseltine, 131; Hurd, 56. That left Major two short of outright victory, but Heseltine and Hurd at once conceded defeat.

Major led the party to victory in the 1992 general election, but his credibility never recovered from the debacle of Black Wednesday, in September of that year, when the pound was forced out of the Exchange Rate Mechanism.

By the summer of 1995, he was so fed up with the attacks on him by the Eurosceptics in his own ranks that he resigned and told them to “put up or shut up”. Michael Portillo’s supporters installed some telephone lines, but Portillo declined to put up. John Redwood did put up, but got only 89 votes to the Prime Minister’s 218.

The final leadership contest under these rules took place following Major’s immediate resignation after the 1997 general election, in which the Tories lost 171 seats, leaving an electorate of only 165 MPs.

In the first round of voting, Kenneth Clarke got 49 votes, William Hague 41, Redwood 27, Peter Lilley 24 and Michael Howard 23. Howard was eliminated and Lilley stood down.

In the second round, Clarke got 64, Hague 62 and Redwood 38. The latter now tried and failed to transfer his support en bloc to Clarke, and Hague won the third round by 90 to 72 votes.

As party leader, Hague introduced reforms to the election rules to make them more democratic. From now on, MPs would whittle the contenders down to the last two, between whom party members would choose.

These rules, and the elections held under them, have recently been outlined in a briefing by the House of Commons Library. Even if one has lived through this period and taken a close or even morbid interest in the successive contests for the Tory leadership, it can be difficult to remember what happened, so here is a summary.

In the first round of voting in 2001, Portillo got 49 votes from MPs, Iain Duncan Smith 39, Clarke 36, Michael Ancram 21 and David Davis 21.

By the third round, Clarke was on 59, Duncan Smith on 54 and Portillo had 53, so was eliminated.

The members proceeded to give Duncan Smith victory by 155,933 to Clarke’s 100,864 votes.

In 2003, 25 MPs requested a vote of confidence in Duncan Smith’s leadership, which he lost by 75 to 90. In the subsequent leadership election, only Michael Howard was nominated, so he was elected unopposed.

In 2005, Howard made an outrageous attempt to exclude Tory members from their decisive role in the electoral process, by changing the rules in order to return the final decision to MPs. This was beaten off by determined opposition led by Tim Montgomerie, founder in that year of ConservativeHome.

In the first round of the 2005 leadership contest, Davis received 62 votes, David Cameron 56, Liam Fox 42 and Clarke 38.

In the second round Cameron got 90, Davis 57 and Fox 51.

The members gave Cameron victory by 134,446 to Davis’s 64,398 votes.

There was no further contest until 2016, when in the first round Theresa May got 165 votes, Andrea Leadsom 66, Michael Gove 48, Stephen Crabb 34 and Fox 16. Fox was eliminated and Crabb dropped out.

In the second round, May polled 199, Leadsom 84 and Gove 46. Gove was eliminated and Leadsom dropped out, so the members never got to vote.

While writing this article, I needed to check a quotation, and happened on a piece by Ferdinand Mount, written just after Cameron had overtaken the favourite, Davis, in the 2005 race:

“what is different about this startling result is that Cameron looks like being the first Tory leader to be chosen, primarily and deliberately, because his electors – both Tory MPs and party activists – think he is the man that the public at large would prefer. So far at least, they are putting popularity before ideological soundness.

“Sounds obvious, especially after three thumping defeats. But look back over the past half-century. If popularity with the public had been the criterion, not a single one of the party’s leaders would have made it. Instead, the Conservatives would have been led successively by Rab Butler, Reggie Maudling, Willie Whitelaw, Michael Heseltine and Ken Clarke.”

In the present race, as on previous occasions, MPs and activists will have to decide whether to put popularity before ideological soundness. The result will probably turn on which of those qualities they regard as more important.