Congratulations to Robert Jenrick, the new MP for Newark.

Standing as the successor candidate to someone forced out of parliament amid a high profile scandal is a sticky wicket to say the least. Initially, it seemed plausible that the Mercer effect, plus UKIP’s surge, might be enough to overcome the 2010 majority. As it turned out, the Conservative result last night was more than respectable – a majority of 7,000 votes with a vote share of 45 per cent, down 8.8 percentage points.

UKIP’s much-heralded earthquake rocked for a while in May but Newark seems to have been an aftershock at most – they won 25.9 per cent of the vote, falling short of their best performance in Eastleigh and almost 20 per cent off the outright victory they had once predicted.

Labour barely turned up, with their vote down by 4.7 per cent and securing only third place in a seat that they held not so long ago. Neither their campaign nor their result looked like a party on track for government – while Tony Blair won seats requiring bigger swings than this in the run-up to 1997, Miliband is slipping back.

The Lib Dems lost their deposit and came sixth (again) – further adding to the feeling that rather than four-party politics, they have simply dropped out of the running in many places.

By-elections are notoriously tricky beasts – before Newark, the last time the Conservatives won one while in government was Richmond, 1989, when William Hague was elected. As we’ve covered a number of times on ConHome, the challenge is further complicated by the decline of the party’s grassroots membership. Neither Eastleigh nor Wythenshawe and Sale East were pleasant experiences for the Conservative Party – so how was this time different?

The first thing to note is the dramatic improvement in the by-election machine. Having been caught napping in Eastleigh, where a potential Conservative gain became an embarrassing third place, CCHQ evidently had no intention of being caught out again.

Where Eastleigh saw MPs being bussed in in a panic, this time the operation was considered and pre-planned. All Tory MPs were asked to go to the seat three times, Cabinet Ministers five times, and by all accounts their turnout was impressive. This was the tally on the morning of election day:

I’m told over 100 MPs were on the ground as part of the Get Out The Vote operation yesterday. The Tweetminster feed of Tory MPs was packed with messages from Newark (plus, somewhat surreally, Hugo Swire live-tweeting his official visit to Bhutan), notably many of them accompanied by activists from their own associations.

This was a key difference from previous mobilisations of the Parliamentary Conservative Party. Rather than simply use them as doorstep-fodder, the MPs were used as a way to bring along other activists – as a result, hundreds came to the seat from across the country, encourage by their MP or keen to see if they’d find themselves canvassing alongside Cabinet Ministers. It’s an approach much more reminiscent of the old Lib Dem mantra that elected representatives are a force multiplier, as well as a force in themselves.

The Whips co-ordinated the Parliamentary effort, which was sufficiently strict to require even David Cameron to sign the attendance register.

That was part of a much bigger picture. The numbers of activists on the ground reached a remarkable scale – 500 were brought by the RoadTrip2015 event on Saturday alone, reportedly over 1000 turned up for polling day itself, and hundreds of others have taken part at other stages of the campaign. Candidates, too, were all encouraged to come and join the fight.

The Connect 2015 phone bank offered the opportunity for those of us who couldn’t get to the seat due to distance, affordability or work commitments to take part either from CCHQ or from home. I did so, and it found it a simple process – callers are cleverly egged on by a league table in which you win points for successful calls, introducing an edge of competition into it. Judging from my starting position on the table there were at least 200 people taking part like this, including more than a few advisers and senior Downing Street figures, some of whom made hundreds of calls each.

Ultimately, it was a grassroots effort that the other parties failed to match – made all the more remarkable that it came immediately after most people had spent weeks fighting local and European elections. As well as the failure of the much-tweeted, oft-repeated “Labour Doorstep”, those on the ground report a surprisingly sparse showing of UKIP activists. In the final few days, their most visible presence was purple and yellow vans, loudhailers and billboards rather than canvassing teams.

Second, we were fortunate to have a candidate in place well before the by-election was sprung. Whereas in Wythenshawe and Sale East a candidate had to be selected to fight the by-election from a standing start, following the sudden death of the sitting MP, Robert Jenrick was in place several months before Mercer chose to resign. This meant he already had a certain amount of name recognition, and campaigns on local issues were made more plausible by the fact they existed before a by-election was called. Several of those who door-knocked in the constituency told me they found this contrasted favourably with UKIP’s decision to select Roger Helmer, a full-time politician without an established presence in the seat.

Third, the later stages of the campaign used UKIP’s confidence against them. With the ‘kippers and the media having talked up the possibility of the first purple Westminster election victory, the Conservatives were able to make people think carefully about the seriousness of the decision before them. This wasn’t a “free hit”, as Farage described the European elections, this was a battle over who should be sent to Parliament to represent the people of Newark – were UKIP really the best party to do that?

I heard its effectiveness among wavering Tories for myself, speaking to people on the phone who said they had planned to vote UKIP but thought better of it in the polling booth. This is a foretaste of the battle for the minds of the right in 2015 – will people vote for Farage to hit at Cameron and the political class, or will they vote Conservative to deliver better government and keep Labour out? When there are no “free hits”, it seems voters are more practical and less emotional.

There are also some suggestions that the “stop UKIP” message succeeded in turning Liberal Democrat and Labour voters blue – which would be the first example of Farage’s high media profile backfiring on him. It’s a distinct possibility that UKIP are succeeding in detoxifying the Conservatives in a way that David Cameron could not do, bringing new voters into the fold who would not previously have considered voting Tory.

We should celebrate the election of Robert Jenrick, and the end to a 25-year run of by-election defeats when in Government. We should certainly celebrate the evident improvements in the by-election machine. But we should also carefully note the lessons.

Select early: Having candidates in place well before election time brings benefits at the ballot box. That applies as much to General Elections as it does to by-elections. As such, the continued delay in selecting PPCs for hundreds of seats is troubling – each week that goes by is one less week for them to campaign and build name-recognition for 2015. Plus, you never know when another by-election might crop up, as Wythenshawe showed

Develop an effective message against UKIP: Working out how to deal with any new threat takes time and experimentation. Sure, Farage has changed his tune now to talking about the achievement of coming second, but never forget this was a seat that he at one point seriously hoped to win. Even a few hours before the count finished he was talking about a close-run thing and a majority of no more than 2,500. Newark was a success, and we should study why.  Next year we will need to repeat the feat in a much bigger battle.

The grassroots are essential: This by-election was ultimately won by the brute force of a massive activist turnout – mobilised by MPs, bolstered by the crack troops of Team 2015 and RoadTrip2015. Those involved should be proud – it was a vivid demonstration of the importance of a party’s grassroots. Fighting and winning a General Election will be much harder than holding on to Newark – and the number of fronts on which to fight will be vastly greater. The only way to do so is to grow our supporter base.

An air war, press releases and speeches on TV are not enough in themselves to win elections – you need activists to deliver the message face to face. A party that neglects or disdains those people is one that loses. At times the Conservative Party has forgotten or even dismissed that idea, to disastrous effect. Newark is, I hope, a sign that it is starting to remember and embraced it.