This is essentially what people are afraid of. As you suggest, it may well be that the tired “they’re taking our jobs” argument is simply a smokescreen for concerns about the loss of culture implicit in a period of mass migration. Perhaps people are becoming culturally protectionist and don’t want to be branded “racist,” so are making up economic nonsense to cover their real concerns.
It would be interesting to know if people would hold the same concern if the source of the migration crisis was the predominantly Buddhist countries in South-East Asia or religiously neutral Japan. Part of me thinks that Middle-Easterners are taking heat from the far right for the same reason that Asian immigrants have been in decades gone by — the anti-immigration mob just doesn’t like immigrants and will come up with any excuse to justify it. But there seems to be something ideologically different about the current wave of migrants around which people’s concerns revolve.