There has been much reflection about how tricky it will be for Michael Gove, the Lord Chancellor, to secure the repeal of the Human Rights Act. He has been offered plenty of friendly warnings that it would be best tp abandon the venture on grounds of insurmountable “practical difficulties” – although these warnings oddly tend to come from those who happen to be enthusiasts for the Human Rights Act on principle.

Then there is the Parliamentary arithmetic. Labour are likely to oppose the repeal – even those most Labour voters would welcome the demise of the HRA. Even before resolving who their new leader should be Labour has decided to give way on the case for lowering the benefits cap and holding an in/out EU referendum. But on the HRA they are unyielding. They are keen to stick to an unpopular policy stance.

This is a mistake by the Labour Party. The European Court of Human Rights has engaged in “judicial activism” – it has decided to impose laws on Parliament. Thus we have had foreign judges who feel they should be able to stop suspected terrorists escaping extradition or enable illegal migrants evade repatriation or to insist that prisoners been given the vote. Accepting this is both unpatriotic and undemocratic – even if you happen to agree with particular ECHR rulings. Yet Labour and the Lib Dems – as well as the SNP who make no claim to British patriotism – are championing the HRA which casts aside Parliamentary sovereignty.

Then there will be the Tory rebels. These days David Davis is always keen to make trouble for the Government. His chum Andrew Mitchell also has personal grievances. There’s Ken Clarke, of course. Dominic Grieve’s objections strike me as more sincere. Any more? Probably. Any proposal that threatens the livelihood of lawyers will always face some difficulty in Parliament.

Yikes! We’ve only got an overall majority of 12. (Or 16 if you consider that Sinn Fein don’t turn up).

However would not any rebellion be offset? Would UKIP’s representative Douglas Carswell wish to maintain the Human Rights Act? Naturally if the HRA is survives that would be a gift to UKIP with could proclaim that it shows how the Conservatives are a bunch of softies who can’t be trusted. But wouldn’t that be rather a confused message if UKIP’s sole Commons rep was one of the culprits who thwarted the disposal of this Blairite legacy?

Among the more pointed objections to the HRA is that it has thwarted the deportation of terrorists. Would the two Ulster Unionist MPs or the eight Democratic Unionist MPs really want to keep it on the statute book? For that to be remembered as the key difference they made in this Parliament? With the DUP in particular they might be reluctant togive their votes away when they might hope for some extra public spending on Northern Ireland in return. Even so…

Then there will be the sheer volume of resistance from the “human rights” industry. It’s already in full swing – and however muddled and dishonest the case being put forward it is certainly vigorous. All those thousands of identical emails flooding into MPs inboxes.

As Charles Moore writes in The Spectator this week:

“Amnesty International and others have placed a large newspaper advertisement telling Michael Gove ‘Don’t Scrap Our Human Rights’. The ad asserts that ‘A government cannot give human rights or take them away’, which, if true, makes one wonder how it can scrap them. Human rights are philosophically a confused idea; but their political power consists in the fact that anyone questioning them can be made to look nasty. People who love making new laws — particularly new laws that cost money — therefore like to present these laws as human rights. Article 29 of the EU’s Charter of Fundamental Rights, for example, says ‘Everyone has the right of access to a free placement service’. Such access may well be a good thing (though I confess to being vague about what a free placement service exactly is), but in what sense is it a human right? Whatever it is, it can only exist because governments legislate for it and pay for it. There is no state of nature in which people have the right to a free placement service. Rights, surely, have practical meaning only if they are justiciable. If they are justiciable, it is important under which jurisdiction justice should be done. Mr Gove, I think, would prefer British jurisdiction to a court in Strasbourg. No human right is ‘scrapped’ by taking this view.”

My hunch is that after all the sound and fury Mr Gove will get the scrapping of the HRA through. The process will provide great drama to the delight of the Westminster Village.

However that may not be the most important matter for Mr Gove over the next five years. I hope that will come with the transformation of the prison system.

Mr Gove’s predecessor Chris Grayling has already delivered privatisation of the rehabilitation service which is set to deliver dramatic improvements.

But what of the way inmates are treated before they leave prison? Some still equate the standard of service with money. This is nonsense. Policy Exchange has highlighted the wide variation in the cost of prison places – from £13,200 to £108,000 a year. Often these would be for the same type of service for the same type of prisoner with scant evidence that spending vastly more achieves better results. For that matter we could save even more money by exporting our prisoners.

This is not to say that overcrowding with prisoners “banged up” for 22 or 23 hours day are imaginary problems. They are all too real. The mistake is to presume they are about the money available rather than the way prisons are organised.

By contrast some of the improvements would cost little or nothing.

Diet might seem a trivial aspect of penal policy but I don’t think it is. Theodore Dalrymple says that giving young offenders at Feltham vitamin pills “reduced disciplinary problems very considerably”.

Then there is the crucial matter of enhancing literacy standards among inmates. I don’t think this is any better than it was in 2010. The perverse incentives are still in  place. The Prisoners Officers Association remain dominant running the prisons for their own convenience – shutting out the Big Society of volunteers who could play such a constructive part.

When Michael Gove was Education Secretary he sought to boost the adoption rate and reduce the number of children in care. Had he succeeded he would have made his current job rather easier. Children in care are more likely to end up in prison than in university. Unfortunately progress proved derisory. It continues to be a genuine form of institutional racism.

Thus the conveyor belt continues unimpeded – with children in care growing up to be inmates in prison.

Mr Gove has formidable talents. Effective penal reform will be harder than drafting repeal of the HRA. His efforts at the former will attract little attention as lobby correspondents obsess over the rebellions and intrigue involving the latter.

However if having already sorted out our schools that Mr Gove can now sort out our prisons that will be a formidable legacy.