It is now 187 years since Macaulay said of Parliament, in one of the great essays which he contributed to the Edinburgh Review: “The gallery in which the reporters sit has become a fourth estate of the realm.”

Since that dangerous year, 1828, when pressure was building for Catholic emancipation and parliamentary reform, the means by which the fourth estate exerts its influence on politics have changed a bit. But any free society derives a great part of its vitality from a press which detects, expresses and gives currency to abuses, scandals, protests, demands for redress, changing currents of public opinion: forces which the politicians must harness or otherwise deal with if they themselves are not to be swept aside.

Which makes the smash suffered by the media on election night all the more striking. The commentators were shown to be out of touch with public opinion, and so were the news editors. News coverage was predicated on the belief that there would be another coalition of some sort, and vast amounts of space and expertise were devoted to working out what sort of coalition it would be.

To say the media almost got it right is no defence. Suppose Aston Villa this afternoon beat Arsenal one-nil. It will be no use at all for those (including myself) who expect Arsenal to win to say we nearly got it right. A narrow victory is still a victory. Villa will still have won the FA Cup.

In a column written a few days after the election, Boris Johnson rejoiced in “the wholesale and hilarious humiliation of the pundits”. He was right: to see such self-importance exploded brought roars of laughter on sofas across the land. The main qualification for becoming a pundit often seems to be the ability to take one’s own opinions more seriously than they deserve to be taken. Election night was a joyful experience for all those of us who are fed up with the pretence of these soothsayers to understand the spirit of the times better than the rest of us do.

What went wrong for the soothsayers? Some at least of them suffered from a kind of incredulity. They could not believe the ruling class was capable of regenerating itself. They assumed that the Conservative Party could not cope with the modern world. It was obviously unable to recruit women or members of ethnic minorities in sufficient numbers. The fact that it had placed its fortunes in the hands of a man who went to Eton, and belonged to the Bullingdon Club, confirmed that it was out of touch.

The people who were out of touch were the pundits. It did not occur to them that going to a good school might be an advantage. Nor could they see that to most voters, the question of whether the Prime Minister was any good at his job was far more important than where he was educated or what clubs he belonged to. Oddly enough, this was an area where the polls gave a pretty good indication of public opinion: they confirmed that David Cameron was more respected than Ed Miliband. But the obvious implication of this preference – that in the secrecy of the polling booth, people might well opt for Cameron – was somehow discounted.

So was the even more unfashionable possibility that some voters were actually rather pleased to have a Prime Minister from a patrician background. The other day I met an economist who claimed that working-class women, such as his own mother, approved of Cameron’s manners. I cannot pretend I have done any research to verify this finding. But it does appear that pundits who went to Oxford are more neurotic about class than most voters are. And it is also evident that many millions of voters are small-c conservatives, who regard it as a matter of common prudence to vote for a party which believes in such staid things as hard work and sound money.

Let us not be too hard on the pundits. Part of the trouble is that they are overworked. The days when a pundit could survive by writing one piece a week are gone. In the age of the internet, they find themselves obliged to pontificate day in day out from early in the morning until late at night. This means they have no time to think or read. Nor do they have time to find anything out by talking to people, perhaps while taking a glass or two of refreshment: they are obliged to subcontract that side of the business to polling companies. We live in the age of the idle workaholic, who hopes the huge volume of his output will obscure the laziness of his thinking and the dreariness of his style.

One does not wish to idealise the past. There have always been pundits who got things wrong. But the one-piece-a-week system had considerable advantages. It gave the writer an incentive to try, on that one occasion, to do his best work, and perhaps even produce something that was not just an echo of what everyone else was doing. If he did a feeble piece, he would have to wait another seven days to clear his name, and might find himself the object of derision, or at least of a wounding lack of respect, in whatever pub or club he frequented. (I write “he” because the genuinely self-important pundit is much more often male than female.)

The humiliation of the pundits has political consequences. On no subject do they feel readier to pontificate than Europe. Here is a field where their expertise, refined on occasional visits to the Continent though seldom by actual residence there, is unrivalled. Their superior knowledge has often enabled them to know when our European policy is going wrong.

What weight do their warnings now carry? David Cameron has already said, with reference to the negative coverage the press will sometimes give to his European negotiations over the next couple of years: “My advice would be – a bit like the election, really – wait for the result!”

The same maxim can be applied to every other field of policy. The Government can hope to cut out the middle man – the pundit – and seek direct approval from the public for its welfare reforms, or its measures to raise standards in schools.

Tony Blair flattered the press by seeking and obtaining its approval. Cameron is now in a position of greater strength than that. He can seek to do what is right for the country, and wait for the press to catch up.