Lewis Baston is author of Reggie: The Life of Reginald Maudling and several books about British general elections. He is a consultant on politics, elections and constituencies.
The Clacton by-election on 9 October will be a very unusual event. Douglas Carswell is one of only four MPs since 1945 to have changed allegiance and then asked the electorate their approval in a by-election.
Leaving aside the Haltemprice & Howden by-election of 2008,  which marked a kind of internal defection by David Davis from Shadow Cabinet Tory to Awkward Squad Tory, the previous MP to do so was Bruce Douglas-Mann, the Labour MP for Mitcham & Morden who joined the SDP in December 1981 and resigned his seat in May 1982. The circumstances were unique, as Douglas-Mann explained in a Personal Statement to the House:
“There are many precedents of Members who have crossed the Floor of the House without resigning. Perhaps the late Sir Winston Churchill is the most famous example. Whether or not one accepts that such a fundamental change as crossing the Floor of the House involves an obligation to seek re-election, I believe that there is none upon hon. Members who consider that their views have not changed fundamentally but who personally feel that their parties have adopted a radically different position since the last general election. That is the position of my colleagues in the SDP.”
My position is different, because I have given specific assurances to the Mitcham and Morden constituency Labour Party, which I have repeated at public meetings, that if ever I were to leave the Labour Party I should resign my seat and contest a by-election.”
Although Douglas-Mann emphasised his personal promise, his decision to call a by-election did not go down very well with his SDP colleagues, some of whom felt that the precedent put them in a poor light. Douglas-Mann was unfortunate, in that if he had resigned the seat and triggered the by-election immediately he might well have been able to get re-elected, alongside Roy Jenkins who gained Glasgow Hillhead in March 1982.
But he ran headlong into the Falklands War, and the local elections two days after his resignation showed the Conservatives well ahead both nationally and in Mitcham & Morden. He lost the June by-election to Angela Rumbold, who scored the only Government gain in a by-election in the last half-century. Douglas-Mann fought the seat again in 1983 and 1987 but to no avail, and he did not become one of his party’s peers.
Before Douglas-Mann, there was another ex-Labour social democrat, Dick Taverne, in Lincoln in 1973; Taverne had fallen out with the Labour Party nationally and locally, particularly on the issue of Britain in Europe which he strongly supported. Taverne’s switch was a triumphant success. He was re-elected in Lincoln as ‘Democratic Labour’ with a 13,000 majority (nearly 35 per cent of the vote, comparable to what the polls have shown so far for Carswell in Clacton).
Taverne also held his seat, although narrowly, in the next general election in February 1974 but was defeated by Margaret Jackson (later Beckett), the official Labour candidate, in the October election. He joined the SDP and fought another by-election, this time in Peckham, only to be defeated by Harriet Harman, another future member of Blair’s Cabinet. Taverne became a Liberal Democrat member of the House of Lords in 1996 and is still alive and well; he recently published his memoirs.
Next in our relatively short list of MPs who triggered by-elections on their change of allegiance comes the case of Sir Richard Acland, the former Liberal, former Common Wealth and former Labour MP who gave up his seat at Gravesend in March 1955 to campaign against the development of the hydrogen bomb. His intention to fight a by-election on the nuclear issue was thwarted by the calling of the 1955 general election. Acland won 13.9 per cent of the vote, not bad for an Independent at the time, but still came a poor third.
With Acland, we exhaust the post-war supply of by-elections called as a result of defection. To this small number, we might add a few cases where an MP switched allegiance late in their parliamentary term, sufficiently close to the general election for there to be no significant gap between the defection and the constituency having a vote. Sir George Gardiner (Conservative to Referendum Party, March 1997) fought the seat, very unsuccessfully, under his new colours in the general election. Christopher Mayhew (Labour to Liberal, July 1974) left his Woolwich East constituency and fought and lost Bath for the Liberals in October 1974, and became a Liberal peer in 1981.
Between 1924 and 1939, there was the isolated case of the Conservative MP Kitty, Duchess of Atholl, who resigned and fought her Kinross & West Perthshire seat as an independent anti-appeasement candidate in 1938, who lost; and a small flurry in the late 1920s triggered by defections from Liberal to Labour. These were William Jowitt in 1929, Joseph Kenworthy in 1926 and William Wedgwood Benn who resigned his seat in Leith in 1927, even though the local Labour Party retained their existing candidate; Benn was elected instead for Aberdeen North in 1928. Leslie Haden-Guest went in the other direction in 1927, from Labour to Constitutionalist, over foreign policy issues but, unlike the others, he was defeated in the by-election. In time he made his peace with the party and was elected for another seat as a Labour MP in 1937.
The number of MPs who have crossed the floor without reference to their constituencies is much larger than the handful who have caused by-elections. There are a few cases in nearly every parliament, although a fair number involve temporary suspension of the party whip. Sometimes whipless rebels form parliamentary groups, which may be wound up when the MPs are readmitted to the fold (as with the anti-EU Conservative ‘whipless wonders’ of 1994-95 and most of the Suez rebels of 1957-58), or continue into contesting the general election as ad hoc small parties (like far-left expellees from Labour in 1991-92 and 1949-50).
By far the largest wave of defections since the confusing free for all of 1914-22 was in the 1979-93 parliament with the formation of the SDP. A few have involved abortive attempts to found new parties, such as Desmond Donnelly in 1968 (Democratic Party), Jim Sillars and James Robertson in 1976 (Scottish Labour Party) and Oswald Mosley and colleagues in 1931 (the New Party, quickly to morph into the British Union of Fascists).
Even so, there is a trickle of people making the leap from one mainstream party to another while remaining an MP (and rather more people elected under one label and subsequently turning up in parliament under a different label after a gap: Churchill’s second defection, back from Liberal to Conservative via ‘Constitutionalist’ in 1924, was of this nature).
These throw up some interesting patterns. During the 1990s and 2000s, the pattern was mostly for Conservatives who found the party increasingly uncongenial to move either to New Labour (Alan Howarth, Peter Temple-Morris, Shaun Woodward, Quentin Davies) or for some ideologically pro-Europe ones to go to the Liberal Democrats (Emma Nicholson, Peter Thurnham, and a number of MPs defeated in 1997 such as Hugh Dykes and Harold Elletson). The realignment of the Conservative Party as Eurosceptic during this period also led to several MEPs jumping ship to the Lib Dems, such as Bill Newton Dunn.
Before the 1980s, most of the defection traffic was away from Labour. Reg Prentice started the 1974-79 parliament as a Labour Cabinet Minister but joined the Conservatives in 1977, following acrimonious battles within his constituency Labour Party. He was famously described (we know not by whom) as having ‘run the four minute mile on the road to Damascus’ for his quick change of colours. He was still very much a Conservative when I spoke to him in 2000, but was very uneasy about the party’s support of the fuel protests of that year because ‘direct action’ of that sort reminded him of what he had grown to dislike in the trade union movement in the 1970s.
Alan Brown, the Labour MP for Tottenham, joined the Conservatives in 1962, and two Labour MPs switched to the Tories in the 1945-50 parliament, Ivor Thomas and Albert Edwards. When a party suffers from repeated defections, it is a sign that its electoral coalition is coming apart – the dramatic losses of the Liberals in the 1920s are the clearest case. The flaking away of the Liberal Party continued even in 1945-51 with its Chief Whip, Tom Horabin, defecting to Labour in 1947 and Lloyd George siblings sitting each side of the House for Labour (Lady Megan) and Conservative (Gwilym).
Winston Churchill, who crossed the floor in 1904 as part of a small wave of Free Trade ex-Tories, reached the Liberal Cabinet and then returned to the Conservatives as Chancellor and Prime Minister, is the patron saint of defectors. As he once mused: ‘anyone can rat, it takes a certain amount of ingenuity to re-rat’.
But re-ratting is not so very unusual; as with most transgressive activities, the first time is the hardest. The political odyssey of Humphrey Berkeley is an example. He was Conservative MP for Lancaster 1959-66, joined Labour in 1971 – fighting the hopeless seat of South Fylde in October 1974 – before joining the SDP and fighting Southend East for them in 1987. He rejoined Labour in 1988. John Horam went from Labour to SDP in 1981, lost in 1983 and was elected for the Conservatives in 1992. In 1994 his first appearance as a Conservative minister at the despatch box aroused a heckle from Dennis Skinner, channelling James Cagney: ‘you dirty double rat!’
Most defectors are relocated to a safe seat at the next general election, ending peculiar interludes such as those that saw Stratford-on-Avon and Witney with Labour MPs and Newham North East represented by a Conservative. Alan Brown tried valiantly to defend Tottenham for the Conservatives in 1964, but in vain. Very few defectors have glittering careers in their new parties, although most find that their chances of ministerial office have improved. Shaun Woodward is exceptional in having reached the Cabinet, something no defector has done since the days of Attlee (whose 1945 Cabinet included ex-Liberals Christopher Addison and William Wedgwood Benn, and Stafford Cripps who had been expelled from Labour during the 1930s), and Churchill, although several others (Howarth, Prentice, the aforementioned Horam, and Lord McNally in the coalition government) have held more junior ministerial office. The life of a defector is often rather lonely; not fully trusted by their new party and ostracised and scorned by their old one. Douglas Carswell is rather fortunate in that even if he is on his own as a UKIP MP, he still has many friends and fellow travellers in his old party.
The decision to defect is usually an excruciatingly difficult one for the MP. There is usually a dark night of the soul before the decision, in which they try to reconcile their inner estrangement with party loyalty. Changing party, for an active politician, is changing part of one’s personal identity. It can ruin careers, friendships, even families, and often attracts harsh personal attacks and imputations of malice or careerism. Enoch Powell, challenged in 1974, can speak for many defectors: ‘Judas was paid! I have made a sacrifice.’
Some of the defectors, particularly those who call by-elections, are people of obstinate, awkward principle (such as Douglas Carswell, Humphrey Berkeley, Richard Acland and William Wedgwood Benn) who can be relied upon to be a maverick whichever party they are in. Some are people, such as Reg Prentice and Alan Howarth, who find their evolving personal values out of step with the direction in which their party is travelling. Some defectors are lost souls. At least two (Alan Brown and Desmond Donnelly) have committed suicide.
There are some moments in history when it is surprising that there have not been more defections. The Liberal Democrats have got through nearly an entire parliament of coalition with the Conservatives without losing an MP through defection, despite many of its MPs having come up through the party when it was clearly on the centre-left. Another surprise is that so few of the ‘unlikely lads and lasses’ of the big Labour intake of 1997 were politically or psychologically unprepared for what awaited them at Westminster and in government. Only one MP in this category, Paul Marsden of Shrewsbury & Atcham, defected, and even he came back from the Lib Dems before the end of the 2001-05 parliament. It would be an even bigger, and more implausible, achievement for the Conservative Party to hold so firm through the next Parliament.
Dick Taverne Against the Tide (Biteback, 2014)
 And the 1986 by-elections re-contested by Ulster Unionist MPs to protest against the 1985 Anglo-Irish Agreement.