Dominic Raab is MP for Esher & Walton.

Denis Healey, the former Labour Chancellor, once quipped that: ‘the difference between tax avoidance and tax evasion is the thickness of a prison wall’. As the Labour party whips up hysteria over tax avoidance, we should remember that the taxman consistently abuses his unwieldy powers – and it’s the ordinary citizen who pays the price.

Paying your taxes is a civic duty. It’s also the lifeblood of our public services. Her Majesty’s Revenue and Customs (HMRC) estimates that unpaid tax dues cost every home £1,200 each year. So making big business cough up its dues is only right. But, when Ed Balls says customers should not just ask the gardener or handyman for a receipt, but file it away in case it could ever be helpful if HMRC prosecutes them, he’s being honest about Labour’s past record and current instincts.

The latest revelations, charting HSBC’s tax-dodging advice, date back ten years. Since then, there have been a string of lenient ‘sweetheart’ deals between HMRC and big business over unpaid revenue – including letting Goldman Sachs off millions in interest by an out of court settlement the judge dryly observed was ‘not a glorious episode in the history of the Revenue’.

Contrast that with HMRC’s record of treating small businesses. Spot checks have surged since the recession. Since 2008, HMRC tightened rules giving struggling firms more time to pay overwhelming tax dues, which led to a 57 per cent spike in government petitions to wind up businesses. And, in 2009, Labour sanctioned deployment of private sector debt collectors to enforce this hard line. HMRC regularly abuses its power. One small business was hit with a £10,000 fine for failing to notify HMRC of a tweak to its name, even though it had paid all its taxes owed.

The treatment of individual taxpayers is worse still. Horror stories include Alexandra Roberts, a teacher from Dorset, pursued in 2007 for £21,000 of tax credits for failing to disclose she had separated from her husband – despite having evidence and admissions from HMRC that she had notified them. These aren’t isolated human errors. The Tax Adjudicator upheld 92 per cent of complaints in 2014, and criticised HMRC for its poor treatment of vulnerable clients. The amount HMRC has to repay following complaints has rocketed ten-fold since 2003 – topping £4 million last year. For all the understandable zeal about pursuing big business, not enough has been done to check HMRC’s abuse of ordinary working people.

The coalition has made important changes in the drive to target the bigger players. Tolley’s tax handbook doubled in length under Labour, attracting the ruses by crafty financial advisers and multinationals they now decry. The Coalition set up the Office of Tax Simplification to streamline the rules. The loophole allowing foreign buyers to avoid stamp duty on expensive property has been closed. Banks’ ability to avoid tax by off-setting past losses against current profits has been curtailed. Forty further tax reliefs have been scrapped. Likewise, while the number of people convicted of tax evasion fell by a third in Labour’s last three years (they didn’t keep proper records before that), it doubled under this Government. The Coalition has also led the drive for international transparency, securing agreement amongst 90 countries (including Switzerland) to exchange information between tax authorities.

Yet at the same time, HMRC’s powers have grown significantly. The Coalition introduced a new General Anti-Avoidance Rule, aimed at pursuing tax planning that satisfies the letter of the law, but not the spirit. That may be fine, if it’s limited to multinationals and those who can afford accountants and lawyers to navigate the increased uncertainty. But such opaque powers also invite more bullying of small businesses and middle income taxpayers.

HMRC will gain the power to require payment upfront, before it has won a dispute for outstanding tax debts. And HMRC is proposing new powers to take tax dues directly from people’s bank accounts without a court order – which would apply to around 17,000 people each year. Again, this may not trouble big business or those with deep pockets, but it threatens to bankrupt those whose cash-flow can’t take the hit. Given HMRC’s track record of institutional ineptitude – and Labour’s vow to target tradesman and ordinary families – the scope for injustice under a Miliband government is enormous. These new powers should be limited to targeting large corporations and the super-rich.

If we want an end to tax avoidance, we need a simpler tax system. In the meantime, the taxman shouldn’t be allowed to bully the little guy, just because he can’t land a punch on the big boys.