And it's unfortunate that our society reacts this way.  

I wouldn't at all be surprised to see the same people yelling about how the immigrants are "taking our jobs!" also yelling "Those damn immigrants are taking our jobs AND our homes"?

A guy loses his job and often "foreigners" are blamed, whether due to outsourcing or immigration.  If losing that job makes his house get foreclosed on, and then a family of immigrants moves into it, that would likely breed resentment, as unfair as that sounds (and is).