The most senior Conservative in Brussels backs Brexit because he wants an immigration system that is fairer to people from outside the EU.