> Their opinion only matters because you make it matter, by caring about it enough to let it stop you from doing something you want to do.

I disagree. If enough people in your community say that you are unwanted, you will become unwanted.