Charlie Elphicke is MP for Dover and Deal.

Over the last five years the amount of income taxes paid by the super rich has fallen by £1 billion. Meanwhile the income taxes paid by everyone else have risen by £23 billion.

With figures like this, it’s not surprising so many people feel there is one set of rules for the richest and another for everyone else.

This needs to change. If we are going to succeed in building a Britain that works for everyone, it needs to be one where everyone contributes to the cost of running public services. A Britain that doesn’t just work for the Philip Greens of this world – but one that works equally for the thousands of BHS staff who face losing their jobs and pensions while he suns himself on his yacht.

What needs to be done is the subject of a Public Accounts Committee report issued today. Back in 2009, under Gordon Brown’s brief and unhappy premiership, a specialist unit was set up in HMRC to investigate the tax affairs of high-net worth individuals – people who are worth more than £20 million.

Back then these super-rich people paid £4.4 billion in income tax. Fast forward five years and this had fallen to £3.5 billion. Meanwhile the total income tax take rose from £250.1bn to £272.9 billion – a rise of £23 billion.

When I asked Jon Thompson, the chief executive of HMRC, how it could be that the richest were paying less he said: “I could not actually give you an answer to that.” The sad reality is that since the specialist unit was set up, the highest earners are paying less tax while everyone else pays more.

So what’s going on? Firstly the way the unit works needs to change. Each super-rich taxpayer gets a personal “customer relationship manager” from HMRC. These relationship managers actually warn their clients of HMRC’s position before they enter into a tax “activity”.

With this kind of approach, is it any wonder there has only been one successful prosecution for tax dodging in six years? Surveys of tax accountants report they are happiest when dealing with the unit rather than HMRC in general – this is not how it should be.

HMRC should turn the “customer relationship managers” into “taxpayer investigation officers”. Currently they are light years off their stated aim of 100 prosecutions a year. The unit itself costs £15 million a year to run. One third of the super-rich are under investigation. Yet the tax paid by the high net worth individuals has plummeted by some £2 billion in real terms. It’s clear a change of approach is needed.

It’s not as if HMRC aren’t more successful in other areas. Indeed, HMRC prosecute about 1,000 people a year – a quarter of cases involving less than £10,000. They need to do more to take on the big fish, not just the minnows.

We also need to do much more to tackle the promoters of tax avoidance schemes. The National Audit Office found that an eye watering £14 billion of tax is at risk from “marketed schemes” sold by promoters, and £1.4 billion of this was tax avoidance by the super-rich. According to HMRC, there are 3,000 such schemes in use by 105,000 “unique users”.

We need to put an end to the promotion of marketed tax schemes – their activity simply means everyone else will end up having to pay more in tax. We need to take firm action. It should be made a criminal offence to market a tax avoidance scheme.

If we are going to succeed in building a Britain that works for everyone, we need a tax system that works for everyone, not just the privileged few. HMRC deserve credit for setting up a unit focused on tackling tax avoidance.

Yet the results so far have not been what we need to see. The number of high net worth individuals has risen from 5,900 to 6,500. But the amount they’ve paid in income tax is down £1 billion while everyone else is paying so much more.

We must remember that the wealthiest in our society do indeed pay vast amounts in tax. Yet it’s clear there are still people out there who will do anything to avoid taxes. That’s why HMRC’s priority must be to crack down on the super-rich individuals and multinational companies who game the tax system.

They need to know that in doing so they have the support of the Government, Parliament and the people in making sure everyone contributes to the cost of running public services.