You have to read the analysis below as well as look at the charts. From the articles I linked
As a further way of characterising countries, we include a second measure based on the percentage of people saying that immigration ‘makes the country a worse place to live’ On this measure, the UK maintains a similar rank position as one of the more positive countries in the sample, and similar to Switzerland at 18%.
These two measures can be thought of as capturing opinions on future migration flows and current population stocks. In most of these 13 countries, it appears that people are more negative towards the idea of continuing flows than about the immigrants already present. Finland, for example, is a country where 42% of the public would prefer few/no immigrants of another race coming to live there, whilst, at the same time, just 19% think immigrants make the country a worse place to live.
It is still much better than many other countries in Europe.
This isn't a small minority and I can guarantee you the distribution of these attitudes isn't equal between leavers and remainers.
Ok sure. I probably shouldn't have said minority. Yeah of course the distribution isn't going to be equal. However people pretend it was all about racism when it clearly wasn't.