What allowed me to change and prosper was the freedom to grow apart and lose touch with people. It’s hard to change yourself if you’re stuck in the same social orbit. There’s a gravitational force that pulls you into repeating the same circular pattern over and over again. Breaking out of that takes tremendous force.
This is a variation on the old saw that “your right to swing your fist ends where my nose begins.” We often forget (or ignore) that no right is absolute, because one person’s rights can conflict with another’s. This is why freedom of speech doesn’t protect extortion, and the right to bear arms doesn’t license armed robbery. Nor is this limited to rights involving the state; people can interfere with each other’s rights with no government involved, as when people use harassment to suppress other people’s speech. While both sides of that example say they are “exercising their free speech,” one of them is using their speech to prevent the other’s: these are not equivalent. The balance of rights has the structure of a peace treaty.