It may take time to not care what other ppl think.
I think as a kid, I cared too much and that was my demise.
You should also understand that there may be some perceived bias and it could be all in your head when you see the stares and mouths moving.
I also like to be alone for the reason of getting away from herd like mentality. When you belong to a certain group and respect that group, it’ll be hard to not allow them to influence your thinking. Peer pressure is a crude mistress but peer pressure from society is a mobile jail sentence.
Free your mind and think on your own. I find it quite refreshing and liberating.