Lord Green is Chairman of MigrationWatch UK and a cross-bench peer.

The Conservatives seem to be having a nervous breakdown over immigration as, yet again, the race card is being played with the clear intention of closing down the debate on immigration. As for the Windrush affair, it was certainly a serious mistake in the execution of a policy designed to reduce illegal overstaying rather than to reduce immigration, but the row led to the Home Secretary attacking her own staff in unprecedented and, frankly, indefensible terms.

She was followed by Messrs Johnson and Gove declaring their undying attachment to a liberal immigration policy. Nothing to do with numbers, they tell us. Actually, it is. Numbers are at the heart of the issue. A YouGov poll last August found that 41 per cent of UK voters saw immigration as the top priority in the Brexit negotiations. Furthermore, a poll for the Migration Observatory in November 2016 found that three quarters of the public want to see reductions in immigration, including 50 per cent who want them to be significant.

Now we have learned, over the weekend, that our chief Brexit negotiator has told his EU counterparts that we are prepared to negotiate on a “labour mobility and immigration regime”. Not a word of warning or explanation to the British public, yet this marks a move onto what could prove to be extremely dangerous ground.

It need not be thus. There is a way forward, but it requires the Government to set out now the immigration framework that they envisage after Brexit. They have been dithering for months about when to do so. At present the White Paper has been put off until the late autumn, yet without that firm base we risk being pushed around by the EU Commission, as we have been since the outset of these negotiations.

The key will be to focus on immigration for work and, especially, on the less highly-skilled. The crucial number is that, over the past ten years, 80 per cent of EU migrants for work would not have qualified for a non-EU work permit. It follows that the simplest and best solution would be to incorporate EU workers in the existing ‘Tier 2’ work permit scheme. This regime, in particular the salary threshold of £30,000 a year, would cut out most of the lower-skilled EU nationals who might wish to work in the UK. Whether or not a quota is also desirable is an open question; it might be necessary both to be seen to treat EU and non-EU workers on a similar basis while also allowing a negotiation on its size. Access to Britain on the basis of a job offer only, as some have suggested, would be no control at all – it would simply be a bonanza for employment agencies.

Any proposal for work permits is bound to trigger intense pressures from those industries that have benefitted hugely from a virtually unlimited pool of competent workers willing to work for very low pay. The Government has handed this hot potato to their Migration Advisory Committee for advice but that, too, will not be available until the autumn.

There may also be a need for lower-skilled EU staff such as construction workers, for whom a temporary scheme of three-year non-renewable visas would allow time for British workers to be trained.

Another special case is agricultural workers. There could be short-term admission for agricultural workers who would be admitted for a maximum of six months, with no benefits but with a guarantee of the minimum wage. Employers should be required to register and ensure the departure of their workers.

It is urgent that we should have such an outline of an effective framework for our future immigration system. It could and should be presented to the EU in a positive way. Both sides will want to see ease of access for tourists and visitors, including business visitors. It would also be very much in all our interests to encourage young Europeans to experience life in the UK for a while. This could be achieved by expanding the present Youth Mobility Scheme which applies to Australians and others, and allows young adults to come to Britain and work for up to two years but with no possibility of extension and no access to the benefit system. It would also be right to introduce measures to encourage EU students to study in British universities.

I return to my central point that the key will be numbers and the risk is that, in a negotiation linked to trade, concessions will be made that will be unacceptable to a wide spectrum of public opinion. Whatever Johnson and Gove may now say, “control” meant reduction for a very large number of Brexit voters. This is especially true of those who are also Conservatives. The economic liberals in the Government may feel that, given the present state of both the Labour Party and UKIP, their supporters have nowhere else to go. They do. They could well stay at home at the next election.