In an ideal world it would go without saying, but our world falls short of that standard, so I’ll say it: the racist attacks widely reported in recent days are utterly disgusting. Thugs yelling abuse at people in the street, poison pen writers slipping vile notes through letterboxes, people in cafes and children in schools telling Poles and Romanians to “go home” – any and all of it is vile, hateful and must be defeated.

As is the way, people rush to identify a trend and then to explain it by allocating a general cause. Sometimes they’re right, sometimes they’re journalists doing so because a trend makes a story more interesting and long-lived, and sometimes, sadly, people seek to try to prove a view which they already hold. Often it may be a mix of all three.

In this instance, it is all apparently to do with the EU referendum – so much so that the hashtag #PostRefRacism has been established.

Is that true, or fair? In some cases, it certainly seems that racists have seized on the referendum as an opportunity to put across their poisonous views. Some of them may well believe – wrongly – that the phenomenon of 17.4 million people voting to leave the EU is a popular expression of support for those opinions. Neither of those things can – or should – be denied.

However, before accepting the most extreme implication of this theory – that such incidents demonstrate that a vote to Leave was in itself a bad thing or, even worse, that Leave voters are therefore racists – it’s worth looking at what we actually know, rather than what we might assume. By definition, each instance of #PostRefRacism has taken place since the referendum. That’s the point of the hashtag. But the reality isn’t as straightforward as attributing the existence of all racism, or every incident of its expression, to the referendum. Correlation, after all, is not the same as causation.

Take, for example, the now-famous photo of four National Front knuckle-draggers at the Monument in Newcastle, my home town, with a banner reading “Stop immigration/Start repatriation”. They were out with their banner and their rather damp flag two days after the referendum. It’s entirely possible they felt emboldened by the result. However, we should also consider that, sadly, Newcastle city centre has been a site of such fascist appearances for some months. The UK branch of PEGIDA, the German EDL-style organisation, hoped to use the city as a recruiting base – indeed, 400 of its supporters converged there back in February at the organisation’s first public event on British soil. They have, happily, found the area to be largely unfriendly territory. The National Front, similarly, has tried to struggle on in Tyneside since the 1970s – in 2011, it stood in the North East for the European Parliament, reaping a mere 640 votes.

This isn’t to say that we should’t be disgusted by those four damp skinheads on a rainy Saturday. But it is a reminder that racism – in the form of them in particular, but also more generally – existed in the UK before last Thursday. We shouldn’t be surprised that it still exists a week later – and we certainly shouldn’t use it to invalidate the perfectly legitimate views on the EU expressed by the majority of voters last week. We should fight it, just as we had a responsibility to fight it before. We shouldn’t commit the injustice of deciding we have 17.4 million racists to fight.

Are we really to believe that the process of having a referendum on the EU made the people featured in these reports into racists? You don’t go from watching an ITV debate with Gisela Stuart to waving a National Front banner or wearing a “send them back” t-shirt in a matter of days. Furthermore, those who call people “p*ki” or post up swastika stickers announcing a “white zone” in Glasgow have by definition a more wide-ranging and deep-seated racism than even the most virulent opposition to migration from Poland or Bulgaria would require.

Nor is it true, as has become fashionable, to attribute the referendum result itself purely to concern about EU immigration. In Lord Ashcroft’s 12,000-voter poll, 49 per cent of Leavers said democracy was their primary reason for voting as they did. Thirty-three per cent said regaining control over immigration and borders was their top motivation. The ComRes poll for the Sunday Mirror found very similar results – 53 per cent of Leavers said “the ability of Britain to make its own laws” was their top reason, with 34 per cent citing immigration. In addition, of course, it would be wrong to suggest that wanting the power to democratically decide Britain’s immigration policy, or to use that power to reduce overall immigration, is in itself racist – or that thinking so makes someone likely to launch racist abuse at a person on a tram, in a bus or on a street.

There’s still a reasonable question, though, about whether the referendum result has made people who harbour such views feel bolder and more entitled to act on them.

In the case of the man in Romford, his “send them back” t-shirt began with the words “Yes! We won!” – he evidently held these views last Wednesday, but it’s undeniable that a few days later he felt justified in hitching them to the referendum result and asserting that it was a victory for him and his ilk. That’s an uncomfortable truth for the vast majority of Eurosceptics who find such views repellent – and, it must be said, it’s a painful reminder as to why we should challenge things from our own side, like Farage’s “Breaking Point” poster, which seek to appeal to such people. As Douglas Carswell wrote recently, such actions are morally indefensible as well as politically counter-productive.

On that theme of uncomfortable truths, Channel 4’s Ciaran Jenkins tweeted on Monday: “It is becoming abundantly clear a certain number of people believe they voted to deport migrants already living in UK.” We may debate what that “certain number” might be – and whether it’s fair to tar all opposition to the EU by association – but there evidently are some people out there who see this as an opportunity to demand exactly that. How should we deal with them?

During the referendum itself, some EU migrants in the UK felt fearful for their future. Would they be allow to stay here after Brexit? British Future, led by Sunder Katwala, asked the major participants in the referendum to commit that EU migrants already resident in the UK would keep that right in the event of a Leave vote. Vote Leave themselves responded extremely clearly: “There would be no question of EU citizens being removed from the UK because of the referendum result.” UKIP did the same: “British Future are right to say present EU citizens will be allowed to stay in UK post Brexit.” All the other major Leave groups held that position, to their credit.

Just because the referendum is now over, though, doesn’t mean we should stop repeating it – if some people are out there claiming, falsely, that this is what the referendum means then we, the vast Leave movement, should be shouting them down. Doing so would be good for the coherence and happiness of our society – that it would help to preserve the reputation of our cause is a side benefit.

Others must acknowledge their responsibility to stamp out this claim, too. Shamefully, while Leave campaigners were happy to make clear that these fears were baseless, the Stronger In campaign was deliberately reluctant to do so. Its head, Will Straw, tweeted ominous rhetorical questions asking “what would happen” to EU migrants living in the UK. He even went on national television to speculate that if we left we would be “sending people home”. He must have known that his opponents were proposing no such thing – Peter Bone pointed it out to him live on air – but was still willing to raise that spectre (though he did later acknowledged in a tweet that “they [Leave campaigners] want to keep existing migrants”).

The Government also played its part in allowing such ideas to persist. Not only did they fail to guarantee existing residents’ rights when pressed to do so by British Future, but in May a Government Minister deliberately left the question unanswered in Parliament.

It is therefore quite rich that some of those from Stronger In and the Government are now wringing their hands about the idea that some people may have thought existing migrants could be sent home after a Leave vote. The Leave campaign was very clear they would not – but prominent members of the Remain campaign deliberately claimed and implied that they might be, and passed up opportunities to reject the idea out of hand. That was dishonest and irresponsible, spreading fear among migrants resident in the UK and encouraging bigots to think their hateful fantasies might come true, all in order to try to win some votes.

Sad to say, the Government still hasn’t fully clarified its view on the issue, even though the campaign itself has ended, preferring to talk only of offering stability for the next two years, leaving an ominous silence hanging over the period beyond that point. By contrast, Boris Johnson was extremely clear on Monday: “EU citizens living in this country will have their rights fully protected, and the same goes for British citizens living in the EU.”

All sides of the EU debate must now confirm this to be the case. We should also hear a lot more, as soon as possible, about exactly what the detailed status and extent of those rights will be. It seems that the issue breaks down largely into three types of right: the right to reside and work, the right to access the UK welfare system and the right to bring dependents to the UK.

The first ought to be entirely uncontested. Those living and working here on referendum day have an acquired right to do so which we must maintain – as the pledges of Leave campaigners, as well as common sense and common decency, would dictate.

The second was already planned to be subject to some changes, as the 2015 Conservative manifesto proposed the requirement to work for four years before gaining access to benefits. That policy will presumably remain the basis for reforms once we leave the EU and regain the power to make such changes without permission from Brussels. If anyone seeking the leadership intends to change it, they should make that clear up front, and present their case to Party members.

The third will, I foresee, prove much more controversial. Voters would be concerned if a back door to the immigration system was opened, but there are also some well-publicised and understandable complaints about the existing, extremely stringent, requirements on non-EU migrants who seek to bring their dependents to the UK. As the leadership race progresses, I would not be surprised if it became a hot topic of debate for the rivals hoping to become the next Prime Minister.