Nicky Morgan is Chair of the Treasury Select Committee, a former Education Secretary, and MP for Loughborough.

On Saturday, the Guardian printed an interview with me.  It was a very fair account of the conversation I had last week with the interviewer. And it included an observation I’ve been making for months – that the oxygen has been sucked out of Whitehall by Brexit, since the best brains in government are now working in our EU-facing departments.

This consequences of this can now be seen in yesterday’s news about the mass resignation of the Social Mobility Commission’s board over its frustration that ‘[the Government] is understandably focused on Brexit and does not seem to have the necessary bandwidth to ensure that the rhetoric of healing social division is matched with the reality’.  Whilst Alan Milburn is not a Conservative supporter, the fact that a Tory peer and a former Education Secretary, Gillian Shephard, has resigned should be a major cause for concern for Downing Street.

Addressing the root causes of social immobility should be on the ‘to do’ list of every Conservative Government.  The Prime Minister stood on the steps of Number Ten on her first day in office, and spoke about supporting the ‘just about managing’. All Conservatives know that ‘aspiration’ is a key watchword for any Conservative-led administration, in local or national Government.

So what has gone wrong? Certainly, Justine Greening has made no secret of her personal commitment to social mobility. The Opportunity Areas that she has launched (which widened the Achieving Excellence Areas that I identified in my own 2016 Education White Paper) are all about tackling one of the major ongoing inequalities in Britain – the fact that, in far too many areas, young people are being let down by their education, and not being given the start that they need in life.

News reports last week confirmed that one of the main subjects Downing Street want MPs to talk about is educational standards. Why on earth has it taken until November 2017 for this to be identified, given the enormous programme and progress in our schools since 2010? One of the answers, surely, is that until last June’s general election, talking about education meant talking about the growth of selective schools. Now we are paying the price.

Had the referendum result been different, and David Cameron not resigned, his government had been planning to turn its attention to launching and putting into action its Life Chances Strategy. This would have built on his speech in January 2016. The plan was to roll out a programme of real social reform aimed at transforming “the life chances of the poorest in our country and offering every child who has had a difficult start the promise of a brighter future”. But in December last year, the new Government confirmed that the Life Chances Strategy had been dropped. This would have been understandable if something new and equally ambitious had been put in its place.

My colleague Robert Halfon talked at one of the ConservativeHome fringe meetings during October’s Party Conference about the Party’s symbol becoming a ladder, to show that the Conservatives were all about offering a ladder of opportunity to everyone in Britain. If this galvanises this Government and the Party into talking about social reform more than Brexit, then let’s make the change fast.

The true cost of Brexit is not the likely “divorce bill”, nor the cost of a slowdown in our economic growth (although both divert money away from policies which will tackle social immobility). Rather, it is the governmental attention that is being taken away from addressing the real reasons why where you are born and your family background still matter far too much in modern Britain.

And for the Conservative Party, a further Brexit cost is that voters now suspect (or in some cases believe) we care more about the EU than we do about them. From now on, surely, every announcement we make has to be motivated by how it will help people get on in life and boost their life chances.