This is the same reasoning many people have when they say they are spiritual, but not religious. (As opposed to those who are agnostic or atheist, which are not the same, of course.)
We can look at almost anything humans do. Take digital photography, for example. There’s no shortage of people quick to make up “rules” of the game. It’s wrong to use editing software, you “have to” shoot raw, etc., etc. Not as serious or impactful, but the same type of thinking.
It’s a function of how the human brain works. We are a product of our ancestors, and knowing who is “them” and who is “us” is how our ancestors stayed alive to have children. And we, too, separate people into “them” and “us” — that’s how our brain works. It’s why we have racism and hate. Them vs us. Religion has not been spared, because every religious group is made of people who have brains that think in terms of them and us.
No, you aren’t wrong for thinking the way you do. I tell my daughter that she needs to follow her own heart, even if I don’t agree with all her decisions. Because it’s HER life. I get to live mine. She gets to live hers. And that applies to you, too. Sorry that was so long winded. :)