If you didn’t see the recent Channel 4 documentary The Secret Life of Prisons, I’d strongly recommend it. Be warned: it isn’t enjoyable, or entertaining. It’s grim as hell – prisoners high on drugs, routine inmate-on-inmate violence, a culture based not just on a callous acceptance of suffering but even enjoyment of watching others suffer.

The fact that much of the film is made up of clips shot by prisoners themselves on illicit mobile phones inside British jails points to a large part of the problem. No-one would be surprised to learn that the criminals convicted to custodial sentences are prone to violence, like to take drugs or are happy to harm others for their own gratification. That’s why many of them are there in the first place. What’s troubling is not that they want to do these things, but that our prison system routinely lets them do so.

Those mobile phones are a tell-tale sign of failures in security and discipline – they’re forbidden, but they’re being smuggled in successfully in large quantities. Their new owners use them to post barefacedly (literally) on social media about the contraband they enjoy. Today’s Mail features pictures of drugs, booze and even hot takeaways that have been brought in.

Such reports highlight the degree to which our prison system fails inmates, victims, taxpayers and wider society. What hope of rehabilitation – and thus of preventing future crimes – can there be if prison is a place where inmates arrive clean and leave with a drug addiction? How do we expect them to prepare for a law-abiding life in wider society when their time in prison is dominated by gangs and violence? What respect for the law is learned in a prison where the rules are routinely broken without consequence?

I’ve written before about the moral, conservative case for reforming prisons to make them places for rehabilitation as well as punishment. At the moment, far too few of our prisons offer either.

Recent escapes and riots, followed by the unlawful decision by the Prison Officers’ Association to instruct its members to stop work earlier this week, have intensified pressure on Liz Truss and her department to demonstrate action.

Some of the commentary, however, has been unfair. After only a few months in the job, she published a White Paper proposing some sizeable changes – including thousands more staff, new prisons and some measures aimed at improving discipline and rehabilitation. The intent is there, even if the timescale needs to be accelerated.

For the POA, the issue seems to be more simple. They say they are underfunded and understaffed, and that’s that. It’s certainly true that they carry out a dangerous job for which they receive little thanks, often due to the fact that their work is inherently not in the public eye. It’s also true that our old Victorian prisons aren’t fit for the job, and simultaneously that replacing them is both difficult, time-consuming and expensive.

But we shouldn’t make the mistake of imagining that these problems can be solved simply by pouring more cash into the current system. The fact is that prisons suffer from failings that are systemic, so at best just throwing more cash at the system would see a sizeable share of the money go to waste.

There are two possible models for a sizeable and challenging reform of the prison system.

The first lies in Theresa May’s years as Home Secretary, when she pursued police reform. She summarised her approach, and the political challenges, in a speech to Reform back in May in which she made clear that her starting assumption was that traditional ways of “reforming” policing didn’t work. She had to improve the transparency and local accountability of the police, but she also had to take on and defeat the claims by what we could call a Blue Blob that anything other than more money would leave to a rise in crime. At times the process was quite confrontational, with some memorable backlash from the Police Federation in particular, but it worked:

“Policing is more diverse, more professional, and better qualified than ever before. Public confidence has been maintained and the proportion of officers on the frontline is up. And crime is down by well over a quarter, according to the independent Crime Survey for England and Wales, even as police budgets have fallen.”

The other model is Michael Gove’s school reforms. When he became Justice Secretary he argued that prisons were less like police forces and more like schools – or rather they ought to be more like schools, if we want successful rehabilitation to take place within them. He delivered the Longford Lecture yesterday and touched on this very topic:

“We should want prisons to have principals running local colleges, business people hiring apprenitces, employers recruiting new colleagues, social work leaders involved in the supervision of those in care and volunteers who want to make a difference all involved in setting goals and providing opportunities.”

Gove’s experience in education reform saw him successfully introduce the ultimate in localism, setting the vast majority of schools free to develop their own service and allowing parents greater choice of where and how their children would be educated. But he, like May, had to take on and defeat vested interests in order to do so. The original Blob, just like the Police Federation, argued that only more money and no challenge to the professionals involved would improve results. They, too, were proved wrong.

All this suggests that Truss is correct to make the case that greater autonomy for prison governors and therefore greater innovation and experimentation is key to changing our prisons from a network of monolithic “criminal warehouses” into a variegated network of institutions which change people’s lives and save others from falling victim to crime.

But it also means that, if she is to succeed, this week’s clash with the POA is unlikely to be the last. Their happiness might be bought by stuffing their mouths with gold and retreating on challenging the old, failed way of doing things, but that would achieve nothing for taxpayers or society.

As well as accustoming the POA to the idea that things are going to have to change, the Government must also make clear that it expects the Association’s help in crushing corruption among prison officers. People don’t like to talk about it, but are we really supposed to believe that all these phones, drugs and even takeaways are getting into supposedly secure prisons without any officers turning a blind eye (for example, by neglecting to spot drones flying over the prison walls) or, worse, actively colluding in the lucrative work of smuggling?

In any system there are outstanding people, a mass of people who work hard but keep their heads down and a minority who are actively bad. All those Prison Officers who have a fervour for radically improving our prisons should be sought out, embraced and empowered to do so. Others who would prefer a quiet life must not be indulged. And the minority who are corrupt must be rooted out.

Last, but not least, there is the question of what happens to inmates after they are released. Imagine a prison system characterised by discipline and rehabilitation, in which a prisoner didn’t encounter violence or drugs during their time inside. When they are let out, there is still an attendant risk that they will reoffend. They are going back to the same place, with the same friends and the same local criminals, they have next to nothing to call their own, they may still be afflicted by serious personal problems and they suddenly have a liberty they have lacked for months or years. Unaided, some will go back to their bad old ways – the probation service can prevent that.

At the moment, we have an artificial division between prison and probation. The two public services happen in different places, but their objective is the same – to keep society safe from further crime. It would be wise to improve the freedom of prison governors to innovate in rehabilitation, and to make them more accountable for their performance, but such a system would be imperfect if their hard work is undermined on release. Quite how to tie the two together into an effective team should be the Government’s next challenge.